Most teams don’t watch their AI cost slowly climb. They open the bill one day and find this month is double last month’s. By then the money is already spent.
This article doesn’t cover budget estimation (already covered in common AI API budget errors) or bill reconciliation (what to do when the bill doesn’t match the pricing page). It covers something different: how to spot trouble before the bill arrives. AI cost runaways usually announce themselves through 7 early signals, each with a specific diagnosis path.
Prerequisite: what data you actually need
Before the signals, one thing must be true: you need usage details, not just a monthly total. Anthropic Console, OpenAI Usage Dashboard, every provider exposes per-day, per-model, per-key breakdowns. If you only look at the end-of-month invoice, 80% of the signals below will hit you too late.
Get at least these three:
- Daily cost curve (split by model)
- Per-API-key attribution (which project / service is using which key)
- Input vs output token ratio (this is where most invisible problems hide)
The 7 signals below are ordered by “severity × how easily it gets missed.”
Signal 1: a sudden spike in the daily cost curve
What it looks like: daily cost was steady at $50, suddenly jumps to $300, then settles back to $80.
Most common causes (ordered by frequency):
- A scheduled job’s retry loop degraded — what should run once daily got stuck running every hour because of a misconfigured cron or retry policy
- A new feature hits a corner case — the new “long document summarization” had no length cap; one user uploaded a 200MB PDF
- Cache got cleared — a deploy wiped Redis and a flood of previously-cached prompts hit the real API
- Client-side retry storm — your server returned a transient 5xx and the client retried blindly
How to diagnose: split by API key first — is one key abnormal? Then split by endpoint — is it /messages or /embeddings? Finally, drill down to hourly resolution — is the spike spread evenly or clustered in one window?
Stop the bleeding: if it’s high-priority, immediately add client-side token-bucket rate limiting (p-limit, bottleneck) before the retry storm continues.
Signal 2: output token ratio creeping up
What it looks like: input/output ratio was 5:1, this week it’s 2:1 or even 1:1. Total cost may not move much, but the impact on the bill is huge — output is typically 3-5× more expensive than input per token.
Why this signal gets missed: everyone watches total cost, not structure. When structure changes, you’re paying the same money for worse content quality.
Common causes:
- Prompt no longer constrains output length — the
Be conciseinstruction got overwritten when someone updated the prompt template - Schema changed — what used to be a flat JSON is now a nested object, and the model dutifully expanded every field
- Switched models — moving to a reasoning model means thinking tokens count, but they don’t always show up cleanly in your logs
- Streaming output saved as full payload — duplicated intermediate frames
Output tokens dominate cost explains why output is the cost driver in detail. A “sudden ratio shift” usually means one of these four causes is in play.
How to diagnose: pull 5-10 of the most expensive requests this week, look at the full prompt and response. You can usually spot the change by eye.
Signal 3: cache hit rate dropping
What it looks like: prompt caching hit rate was 60-70%, drops to 20% one day.
Prompt caching (Anthropic’s cache_control, OpenAI’s prompt cache) is one of your most powerful cost levers. Going from 60% to 20% effectively doubles or triples your cost — you lose the cache discount and pay full price for the full prompt every time.
Common causes:
- One character changed in the system prompt — even a typo fix breaks the cache key
- Variable interpolation crossed cache boundaries — a timestamp got placed before the cacheable prefix
- TTL expired without refresh — Anthropic’s default is 5 minutes, OpenAI varies by model
- Model version switched — every model has its own cache pool
Detailed playbook in prompt caching budget checklist.
Stop the bleeding: check whether your system prompt’s first line contains a timestamp, version string, or any field that changes frequently. Move those fields after the cache boundary.
Signal 4: one API key getting unusually active
What it looks like: you have 5 keys (dev / staging / prod / batch / experimental). Prod usually accounts for 80% of cost. Today, experimental jumps to 30%.
Common causes:
- Forgotten experiment code — data science team’s A/B comparison script kept running
- Key leak — committed to git, posted in an issue, baked into frontend code
- Local script gone rogue — an engineer is running a long loop in a Jupyter notebook and forgot to stop it
The second one is a real emergency — leaked keys can rack up thousands of dollars in 24 hours.
How to diagnose:
1. Provider dashboard → split by API key → identify the suspect
2. git log search for the key value across all branches
3. grep the entire repo (including history) for hardcoded values
4. Check IPs of recent requests against expected originsStop the bleeding: if anything looks off, revoke and reissue. Don’t wait to “investigate further while it’s still running.”
Signal 5: call volume flat, cost rising
What it looks like: total API calls flat or even slightly down, but cost up 30-50%.
Common causes:
- Switched from cheap model to expensive model — GPT-4o-mini → GPT-4o is a 5×+ unit price jump
- Input tokens growing per call — accumulated conversation history is being sent in full every turn
- Attachment volume increased — users started uploading more images/PDFs, single-request token count exploded
- One large customer changed behavior — a single tenant’s conversation length and frequency went up
The second one gets missed most often — total cost looks fine, but cost-per-call is silently inflating.
How to diagnose: track an “average tokens per call” metric. Plot it weekly. Watching this curve catches problems earlier than watching total cost.
Signal 6: test/monitoring environments using disproportionate cost share
What it looks like: dev + staging combined > 15-20% of total cost.
Healthy ratio: dev/staging should be < 10%, prod should dominate. If dev is 30%, you have one of:
- Tests not properly mocked — places that should use fixtures are hitting the real API
- CI pipeline running real LLM tests — every push costs $5
- Local dev not rate-limited — engineers’ hot reloads triggering background calls all day
Stop the bleeding:
- Move CI’s LLM tests to sampled runs (full suite every N PRs, mocks otherwise)
- Switch dev to cheap models (mini / haiku tier)
- Set monthly hard caps on dev/staging keys (OpenAI supports this directly; Anthropic via monitoring)
Signal 7: error rate up but cost not coming down with it
What it looks like: 5xx error rate climbed this week, but cost didn’t drop in proportion.
Normally, a higher error rate should correlate with lower cost — failed requests shouldn’t bill. A few patterns break that:
- Retry logic stacked at multiple layers — SDK auto-retries + app-layer retries + queue retries → one failure becomes three billed attempts
- Wrong fallback strategy — primary model fails → fallback to a more expensive model
- Streaming cancelled but tokens billed — some APIs charge for already-generated tokens even when the stream is interrupted
- Content filter rejected but retry billed — moderation rejection is free, but your retry to a different model is real money
How to diagnose: compute “failed attempts × cost per attempt.” Most teams have never looked at this number directly.
The long game: turn passive discovery into active alerting
7 signals down. The real problem is — you can’t run all 7 checks manually every day. Each needs a continuous metric and threshold:
| Signal | Metric | Suggested alert threshold |
|---|---|---|
| Daily spike | today’s cost vs 7-day mean | > 3× → alert |
| Output ratio | output / input token ratio | > 50% week-over-week increase |
| Cache hit rate | cache hit rate | < 70% of weekly average |
| Key anomaly | per-key daily cost | > 5× that key’s 7-day mean |
| Per-call inflation | avg tokens per call | > 30% week-over-week |
| Test share | (dev + staging) / prod | > 15% |
| Retry cost | retry attempts × unit price | > 5% of total cost |
Implementation depends on team scale — the simple version is a daily email (cron + provider’s cost API + a small script), the advanced version plugs into Datadog/Grafana. The point isn’t fancy tooling, it’s that someone is looking at this data every day.
Triage checklist
If your bill is already running away, the following sequence usually contains the damage within 24 hours:
- Now — set every API key’s monthly cap to 1.5× current month’s spend, prevent further runaway
- Today — work through signals 1-4 to identify the dominant cause
- Tomorrow — deploy client-side token-bucket rate limiting; add per-endpoint rate limits to the most expensive endpoints
- This week — wire up the alerts table above so next month you don’t get blindsided
- Going forward — make monthly cost breakdown a recurring agenda item, not a postmortem
Further reading:
- Avoid these traps from day one of budgeting: common AI API budget errors and how to fix them
- Numbers don’t add up against the pricing page: reconciling AI API bills with pricing
- Concrete cost-reduction levers: 7 actionable ways to reduce AI API cost
- Run your real prompt and projected usage through the token cost calculator for a defensive budget number