Understanding AI API Pricing: Per-Million Tokens, Context & Caching

AI model pricing looks simple on a rate card — a number of dollars per million tokens — but the real cost of a workload is shaped by several variables most comparisons ignore. Get these wrong and a model that looks cheaper on paper can be the most expensive one for your traffic.

This is a practical reference for how the pricing actually works.

Tokens, not characters

Models are billed in tokens — chunks of text, roughly a word or a bit less. A "million tokens" is the standard unit, so you'll see rates written like $2 per 1M input tokens and $8 per 1M output tokens.

Two things to keep straight:

• Input tokens (your prompt) and output tokens (the model's response) are usually priced separately — and output is often several times more expensive than input.
• The same text is not the same number of tokens across providers; tokenizers differ. A "3,000-token prompt" is an approximation, and it won't be identical at two different vendors.

This is why, in the directory, we normalize pricing to USD per 1M input and USD per 1M output — the two numbers that actually let you compare.

Why output tokens dominate the bill

Because output is priced higher and it's what you can't easily cap, a workload that generates long responses can quietly cost far more than the input suggests. Two levers matter here:

• Prompt design. Concise prompts cut input cost directly.
• Max output length. Setting a sensible max_tokens prevents a model from rambling and billing you for verbosity you never needed.

If you're comparing two models for a long-form task, the output rate (not the input rate) is usually the deciding factor.

Context windows are a cost multiplier, not a free feature

The context window is the maximum tokens the model can consider in one call — prompt plus conversation history plus output. A bigger window sounds free, but it's not:

• You pay for every input token in the context, including prior turns you've stuffed in.
• Long conversations that re-send history every turn get expensive fast.

If a provider offers a smaller context at a lower rate and a larger one at a premium rate, the effective cost depends on how much of the window you actually fill. Check the context size in the model index alongside the rate — they're two halves of the same decision.

Batching: the cheapest tokens you'll get

Many providers offer a batch API — you submit requests asynchronously and get a big discount (commonly ~50%) in exchange for slower, non-real-time completion. This is ideal for:

• Backfilling data
• Summarizing large corpora
• Nightly or scheduled jobs that don't need an answer in 300ms

It's wrong for anything user-facing that needs a live response. If a chunk of your workload is batch-able, moving it to the batch endpoint is often the single biggest cost win available.

Caching: paying less for repeated prompts

Several providers support prompt caching — the parts of your prompt that repeat across calls (system prompts, long docs, RAG context) are stored and billed at a steep discount on subsequent hits. This is a big lever for:

• RAG pipelines with a stable knowledge base
• Apps with a long, unchanging system prompt
• Tool-heavy agents that re-send the same scaffolding

Caching changes the shape of your bill from "per request" to "mostly per unique content." If your workload is repetitive, ask whether the model supports caching and what the hit-rate discount is — it can dominate the base rate.

A worked mental model for monthly cost

Rough, but it holds together:

[object Object]

Fill in your token volumes and the real rates (from the provider's pricing page, not a router's headline) and you have a defensible estimate. Then add the router's fee if you're going through one — see choosing a router on transparent pricing.

The bottom line

AI pricing is a function of input vs. output rates, token volume, context usage, and then batching and caching. The rate card is just the starting point. The workloads that win on cost are the ones that batch what they can, cache what repeats, cap what they output, and compare on per-million input/output rates rather than marketing figures.

Related reading

• Free AI API Credits: What They're Worth
• Free Tier vs Pay-As-You-Go: When to Actually Upgrade
• How to Choose an AI API Router in 2026

FAQ

Why are output tokens more expensive than input tokens? Generating tokens is the compute-heavy part — the model produces them one at a time. Reading your prompt is comparatively cheap, so providers price output higher. A long-response workload can therefore cost more on output than input.

Do token counts differ between providers? Yes. Each provider uses its own tokenizer, so the same text can map to a different token count at different vendors. Treat token-based cost comparisons as close, not exact.

What's the difference between a context window and output length? The context window is the total tokens a single call can hold (input history + prompt + output). Output length (max_tokens) is just the part you're billing for as generated text. A large context with a small output cap can keep costs low even for long conversations.

Does batching affect quality? No — the same model and same parameters, just processed off-line. Batching trades latency for price, so it's only for jobs that don't need an immediate answer.