AI cost governance is not another model pricing table. It is the operating system that answers four questions: who is spending, which feature is spending, why did it get more expensive, and what should happen before the bill grows again?
Many teams already know individual tactics: use cheaper models for simple tasks, shorten prompts, cache repeated context, and cap agent loops. The problem is that these tactics do not become reliable until they are tied to budgets, logs, alerts, reviews, and launch gates.
This playbook is for teams that already have AI API usage in production or serious prototypes. You do not need a huge bill to start governance. You only need enough usage that “the AI bill” is no longer self-explanatory.
When you need AI cost governance
A simple spreadsheet is enough when one developer is testing one feature. Governance becomes useful when several features, models, users, or workflows share the same budget.
| Signal | Why it matters | First action |
|---|---|---|
| Multiple features share one API key | Spend cannot be attributed | Split by feature, project, or token |
| You only see the final invoice | You cannot explain growth | Log model, feature, tokens, retries |
| Agents call tools | Loops can multiply spend | Set max steps and tool-call limits |
| RAG adds long context | Retrieved chunks become hidden cost | Cap chunks and input length |
| Users can regenerate freely | Repeated output grows quickly | Add usage limits and idempotency |
| Several models are tested | Selection becomes opinion-based | Track successful task cost |
The goal is not to make every request cheap. The goal is to make every meaningful dollar explainable.
Step 1: record spend attribution fields
A useful AI log should answer more than “which model was used.” At minimum, record these fields:
| Field | Example | Why it matters |
|---|---|---|
| feature | support_answer | Ties spend to product value |
| user_tier | free, pro, internal | Shows who consumes budget |
| model | claude-sonnet, gpt-4o | Compares model choices |
| input_tokens | 4200 | Tracks input cost |
| output_tokens | 900 | Tracks output cost |
| cache_hit | true/false | Measures cache impact |
| retry_count | 0/1/2 | Shows waste from failures |
| tool_calls | 3 | Measures agent expansion |
| status | success/error/fallback | Supports successful task cost |
| estimated_cost | $0.012 | Enables budget monitoring |
Without these fields, cost discussions become guesswork.
Step 2: budget by feature, not by model
A model-level budget such as “$500 for Claude” is too vague. Product teams need feature-level budgets.
Feature budget = request volume × successful task cost × safety factorExample planning table:
| Feature | Daily requests | Successful task cost | Safety factor | Monthly budget |
|---|---|---|---|---|
| Support answers | 5,000 | $0.004 | 1.3 | $780 |
| Document summaries | 600 | $0.035 | 1.5 | $945 |
| Code review | 200 | $0.08 | 1.5 | $720 |
| Agent troubleshooting | 100 | $0.15 | 2.0 | $900 |
“Successful task cost” is the important metric. It includes retries, fallbacks, tool calls, and any extra model calls required to finish the job.
Step 3: define routing rules
Model routing should be simple enough that engineers can explain it.
| Task type | Default model | Upgrade when | Downgrade when |
|---|---|---|---|
| Classification | Low-cost model | Confidence is low | Batch volume is high |
| Summaries | Mid-tier model | Document is long or structured | Only a short abstract is needed |
| Code review | Higher-quality model | Security or core logic is involved | Only formatting is checked |
| RAG answers | Mid-tier model | Sources conflict | It is a simple FAQ |
| Agent tasks | Higher-quality model | Multi-tool reasoning is required | It is read-only analysis |
Routing should live in configuration or code, not in informal team memory.
Step 4: cap RAG, agent, and batch workflows separately
The most dangerous costs are often not normal chat requests.
RAG caps
Set limits for:
- Maximum retrieved chunks
- Maximum chunk length
- Maximum chat history turns
- Whether full documents can enter context
Agent caps
Set limits for:
- Maximum steps
- Maximum tool calls
- Maximum retries
- Human approval for write actions
- Per-task budget ceiling
Batch caps
Set limits for:
- Items per batch
- Concurrent workers
- Retry policy
- Whether expensive models can run low-value tasks
These limits should exist before launch, not after the first budget incident.
Step 5: run a weekly bill review
Every week, answer six questions:
- Which feature spent the most?
- Which feature grew fastest?
- Which model had the highest successful task cost?
- Where did retry rate increase?
- Which prompts produced unusually long output?
- Did cache hit rate match expectations?
Use a short review table:
| Metric | This week | Last week | Change | Next step |
|---|---|---|---|---|
| Total cost | - | - | - | - |
| Successful task cost | - | - | - | - |
| Output token share | - | - | - | - |
| Retry rate | - | - | - | - |
| Cache hit rate | - | - | - | - |
| Avg. agent tool calls | - | - | - | - |
If a metric moves and nobody can explain it, the governance system is incomplete.
Step 6: set alerts and rollback rules
A useful alert leads to a known action.
| Alert | Trigger | Action |
|---|---|---|
| Daily budget warning | 70% of daily budget reached | Downgrade non-critical features |
| Expensive request | Request cost above P95 | Stop generation or trim context |
| Feature spike | Feature cost up 50% week over week | Review recent releases and prompts |
| Retry spike | Retry rate above threshold | Inspect errors and disable loops |
Rollback options:
- Revert to an older prompt
- Use a cheaper default model
- Disable long-output mode
- Limit free-tier usage
- Pause autonomous agent execution
Launch checklist
Before a new AI feature ships, confirm:
- Feature-level budget exists
- Successful task cost was sampled
- Model routing rules are defined
- Input and output limits are configured
- Retries are capped
- User-level quotas exist
- Cost alerts are enabled
- Fallback or downgrade path exists
- First-week review is scheduled
If this checklist is empty, the feature is not cost-ready.
Common mistakes
Mistake 1: choosing by model price only
Cheap models can become expensive if they require repeated retries or human rewriting. Expensive models can be economical when they finish important tasks reliably.
Mistake 2: assuming cache savings before measuring
Prompt caching is valuable, but only if the repeated context actually stays stable and hit rate is high.
Mistake 3: letting agents run without stop conditions
Agents need step limits, tool-call limits, and human approval points. Otherwise one task can become many paid requests.
FAQ
Is AI cost governance only for large teams?
No. Small teams need a lightweight version: feature-level cost logs, budget limits, and a weekly review.
Does governance mean always using cheaper models?
No. It means using the right model for the job, measuring successful task cost, and avoiding waste from retries or unnecessary context.
When does this become AI FinOps?
When AI spend affects pricing, margins, budget planning, or customer-level profitability, you are already doing AI FinOps whether you call it that or not.
Final take
AI cost governance turns model usage from a mystery bill into an operating system. Attribute spend by feature, budget by successful task, cap risky workflows, review the bill every week, and make cost checks part of launch readiness.