What Is a Token in AI? Plain English Guide (2026)
A token is the basic unit every AI model reads, writes, and charges for. Here's how tokenization works across GPT, Claude, Gemini, and every other LLM in production — and what it means for your bill.
A token, in plain English
A token is the basic unit that AI models process. When you send a prompt, the model doesn't see characters or words — it sees a sequence of integers, each one pointing to a piece of text in a fixed dictionary called a vocabulary. The process of breaking your text into those integers is called tokenization.
One token is roughly four characters of English text, or about three-quarters of a word. A short sentence like "How much does AI cost?" breaks down into about six tokens. A full paragraph usually runs 50–100 tokens. A long article can reach several thousand. Code, numbers, and non-English languages often tokenize differently from English prose — sometimes more efficiently, sometimes less.
Characters ÷ 4 gives you a fast token estimate for English.
Words × 1.3 works if you're counting words instead of characters.
Both are rules of thumb. For exact counts across providers, use a real tokenizer — see how to count tokens accurately.
Who this matters for
Developers
If you ship LLM features, token count drives your bill, latency, and whether prompts fit the context window. A 10% over-estimate on a million-token prompt is real money.
Founders
Token cost is the unit economics of every AI product. Picking the wrong model for the wrong workload can 10x your bill overnight — and the difference is invisible until you check.
Power users
If you paste 50k tokens into ChatGPT or Claude daily, the model and context window you pick directly changes output quality, latency, and which features work (long docs, reasoning, code).
How a token actually gets made
Every LLM turns your text into tokens before the model sees them. Four steps happen on the provider's servers — and understanding them explains why your bill looks the way it does.
-
1
Split text into bytes
The tokenizer starts by breaking your prompt into Unicode bytes. Each character becomes a number — this is the safety net that lets tokenizers handle any language, emoji, or symbol.
-
2
Match common subwords from vocabulary
The tokenizer scans for the longest subwords it has in its vocabulary. "Tokenization" might break into "token" + "ization" if "token" is in the vocab but "tokenization" isn't. Common words usually stay whole.
-
3
Assign token IDs
Each matched subword gets replaced with its integer ID from the vocabulary. "How" → 2437, " much" → 1452, " does" → 891. These IDs are what the model actually sees.
-
4
Feed IDs into the model
The sequence of integers goes into the neural network. The model predicts the next token ID, which gets converted back to text. That round-trip happens once per generated token.
Why tokens are the unit of your LLM bill
Tokens are how model providers measure the two resources you consume — and they price each one differently. Here's the full breakdown.
| Token type | What it is | Who pays | Price signal (2026) |
|---|---|---|---|
| Input | The prompt you send | You (full price) | $0.05 – $15 per million (GPT-5 nano to Claude Opus) |
| Output | The model's reply | You (premium) | 3–10× more than input — generation is expensive |
| Cached input | Recurring system prompts the provider recognizes | You (deeply discounted) | 10–50% of input depending on provider (Anthropic 10%, OpenAI/Google ~50%) |
| Reasoning | Internal chain-of-thought the model "thinks" before answering | You (hidden, but charged) | Billed as output — o3 and Claude with extended thinking can blow up costs |
Why the same prompt costs different amounts
Different providers use different tokenizers with different vocabularies. The same English sentence can produce noticeably different token counts — and that changes your bill by 15–25% before pricing even enters the picture.
OpenAI GPT-4o / GPT-5
o200k_base · 199,997 tokens in vocabulary. Code and non-English compress efficiently — often 15–30% cheaper than the older cl100k_base.
Anthropic Claude 4.7
Private tokenizer · ~16,384 tokens. Smaller vocabulary means roughly 30% more tokens for the same text than GPT-4o — and that's a big reason English prompts cost more on Claude.
Google Gemini 2.5
SentencePiece · 256,000 tokens. Trained on 100+ languages — multilingual text compresses especially well.
Things devs get wrong
-
"1 token ≈ 1 word"
Closer to 0.75 words in English. And the ratio swings wildly by language — Chinese often runs 1 char per token, code runs 2–3× more tokens than prose per byte, and emoji can eat 5+ tokens each.
-
"Cached input is free"
It's 10–50% of input depending on provider (Anthropic ~10%, OpenAI and Google ~50%), not zero. Cache misses still pay full price. If your prompt changes by even a few characters, the cache hit rate drops fast.
-
"Reasoning tokens don't count"
They're charged as output tokens. Models like o3 and Claude with extended thinking can generate thousands of internal "thought" tokens you never see — and you pay for every one.
-
"All tokenizers count the same"
See the comparison above. The same sentence can produce 15–25% different token counts across providers. If you're benchmarking costs, the tokenizer matters as much as the price per token.
Common questions
How many tokens is a paragraph?
A typical English paragraph (80–120 words) is roughly 100–150 tokens. A short email (~50 words) is around 60–70 tokens. A 1,000-word article is roughly 1,300 tokens. Code runs 30–50% higher per character than prose.
Is the token count exact?
For OpenAI models, yes — the calculator uses the official tiktoken library and matches what the API bills. For open-weight models (Llama, Mistral, Qwen, DeepSeek), we use Hugging Face Transformers.js, which is ±3% accurate. For Anthropic Claude and closed Google Gemini endpoints, we use a tuned character estimator that's ±15–20%.
Why are input and output priced separately?
Generating text is computationally more expensive than reading it. The model has to predict the next token, then the next, then the next — each step is a full forward pass through the network. Reading a prompt happens once. Writing a response happens many times, so you pay more per output token.
How do I count tokens before sending?
Three methods, in order of accuracy: Tiktoken WASM (exact for OpenAI, runs in browser), HF Transformers.js (±3% for open-weight models), or characters ÷ 4 (rough estimate). See How to Count AI Tokens: 3 Methods That Work for the full breakdown.
What's a context window?
The maximum number of tokens the model can process in a single request — input plus output combined. Claude Sonnet 5 supports 200,000 tokens, Gemini 2.5 Pro supports 1 million. When a conversation exceeds the limit, the provider either truncates oldest messages, returns an error, or charges you for the entire context.
How much does a typical prompt actually cost?
A customer-support chatbot handling 10,000 conversations/month (500 input + 200 output tokens each) runs from a few dollars on Gemini 2.5 Flash-Lite to tens of dollars on Claude Sonnet for the same workload. That's roughly a 10× swing purely from picking a different model — same prompt, same volume, totally different bill. See real pricing across 7 providers for the numbers.
Stop guessing what tokens cost
Paste any text and see real costs across 7 flagship providers in one place. No API key required.
Open Calculator →