The Short Answer
A token is a chunk of text that a language model reads as one unit: often a whole common word together with the space before it, sometimes part of a longer word, a punctuation mark, a group of digits or a piece of an emoji. Models read, limit and bill text in tokens, not in words or characters.
For ordinary English, OpenAI’s rule of thumb is that one token is about four characters or three-quarters of a word, so 100 tokens are roughly 75 words (OpenAI Help Center, “Understanding and counting tokens”, checked on 2026-10-14). The same article stresses that these are estimates: the exact number depends on the model’s encoding, the language and the kind of text. The examples below show how far real text can drift from the rule, and how to get an exact count with WritePadPro’s AI Token Counter.
How Text Becomes Tokens
OpenAI’s current models, including GPT-4o, GPT-4.1, GPT-5 and the o-series, use a byte-pair encoding called o200k_base, with a vocabulary of about 200,000 tokens. Turning text into tokens takes three steps:
- Pre-splitting. A pattern separates the text into words, numbers, punctuation and whitespace. A word usually keeps the space in front of it, which is why “ world” with its leading space is a single token.
- Bytes. Each piece is converted into UTF-8 bytes. An English letter is one byte; an accented letter, a Hindi character or an emoji takes several.
- Merging. Starting from single bytes, the most frequent adjacent pairs are merged step by step, following a fixed ranking learned from training data, until no ranked merge is left. Each remaining chunk is one token with a number (its ID) in the vocabulary.
Common words were merged into single tokens during training. Rare words, names, long numbers and most non-English text were not, so they break into several pieces.
Real Token Counts You Can Check
Every count below was produced with the o200k_base encoding, the same open-source tokenizer the AI Token Counter runs in your browser. The vertical bars show where the token boundaries fall.
| Text | Tokens | How it splits |
|---|---|---|
| hello world | 2 | hello | world |
| Hello, world! | 4 | Hello | , | world | ! |
| The quick brown fox jumps over the lazy dog. | 10 | one token per word, plus the period |
| tokenization | 2 | token | ization |
| unbelievably | 3 | un | bel | ievably |
| WritePadPro | 3 | Write | Pad | Pro |
| antidisestablishmentarianism | 6 | ant | idis | est | ablishment | arian | ism |
| 1234567 | 3 | 123 | 456 | 7 |
| 2026 | 2 | 202 | 6 |
| $1,299.99 | 6 | $ | 1 | , | 299 | . | 99 |
| 2026-11-06 | 6 | 202 | 6 | - | 11 | - | 06 |
| https://www.writepadpro.com/tools/ai-token-counter | 12 | https | :// | www | .write | pad | pro | .com | /tools | / | ai | -token | -counter |
| 👍 | 1 | one token |
| 👍🏽 (with a skin tone) | 3 | the thumb, then two pieces for the skin-tone bytes |
| 🇺🇸 (flag) | 4 | two regional-indicator symbols, two pieces each |
| four spaces | 1 | whitespace runs are often one token |
Two patterns stand out. Numbers are cut into groups of up to three digits, so long IDs, prices and dates cost far more tokens than their length suggests. And anything unusual, such as brand names, rare words, URLs and combined emoji, splits into pieces.
Words vs. Tokens on Real Text
Short examples exaggerate the differences, so we also measured longer text. The body paragraphs of our guide on how to paraphrase without plagiarizing contain 1,029 words of ordinary English prose. In o200k_base they come to 1,284 tokens: 1.25 tokens per word, 4.8 characters per token, or about 800 words per 1,000 tokens. That is a little better than the 750-words rule of thumb, which is fine for planning but not for hard limits.
Code is a different story. One of the JavaScript files behind this site, 12,553 characters long, came to 3,453 tokens, or 3.6 characters per token; a 15,530-character stylesheet came to 4,536 tokens (3.4 characters per token). Code is full of symbols, short identifiers and indentation, and all of them cost tokens.
Language matters even more. Here is the same greeting in seven languages:
| Language | Text | Characters | Tokens |
|---|---|---|---|
| English | Good morning, how are you? | 26 | 7 |
| Spanish | Buenos días, ¿cómo estás? | 25 | 7 |
| German | Guten Morgen, wie geht es dir? | 30 | 9 |
| Russian | Доброе утро, как дела? | 22 | 9 |
| Hindi | सुप्रभात, आप कैसे हैं? | 22 | 10 |
| Japanese | おはようございます、お元気ですか? | 17 | 10 |
| Chinese | 早上好,你好吗? | 8 | 6 |
A single sentence is a tiny sample, so treat these as an illustration rather than a ratio. The pattern is consistent with how tokenizers are built: vocabularies learned mostly from English and code turn common English words into single tokens, while other scripts are split into more pieces per word. Characters per token is meaningless for Chinese and Japanese, where one character can carry a whole word.
Why Claude, Gemini and Llama Count Differently
Every model family has its own tokenizer, so the same text gives a different count in each. Exact counts for Claude and Gemini come from the providers’ own token-counting APIs, which means sending them your text. The AI Token Counter does not upload anything, so for those models it shows a clearly labeled estimate range based on characters per token:
| Model family | Estimate the tool uses |
|---|---|
| OpenAI (GPT-4o, GPT-4.1, GPT-5, o-series) | Exact count with o200k_base |
| Claude | characters ÷ 4 to characters ÷ 3 |
| Gemini | characters ÷ 4.5 to characters ÷ 3.5 |
| Llama 3 and later | characters ÷ 4.5 to characters ÷ 3.5 |
| Mistral | characters ÷ 4 to characters ÷ 3.2 |
Chinese, Japanese and Korean characters are counted separately at 0.8 to 1.5 tokens each, because the characters-per-token rule does not apply to them. The ranges suit ordinary prose; code, tables and long numbers can fall outside them. For billing or hard limits, confirm with the provider.
Context Windows: What Has to Fit
A model’s context window is the maximum number of tokens it can handle in one request. For most models it covers everything at once: the system prompt, the conversation so far, any documents you paste, and the reply. Reasoning models also produce internal reasoning tokens, which OpenAI’s help article notes are billed as output tokens even though you never see them.
Three practical consequences:
- Leave room for the answer. A prompt that fills nearly the whole window leaves almost nothing for the reply.
- Chats grow. Every turn adds the previous messages again, so a long conversation can reach the limit even with short prompts.
- Plain-text counts are a floor. Message formatting, tool definitions, images and files add tokens that a text count does not include.
The AI Token Counter’s table shows how much of each popular model’s window your text would use. For OpenAI models the bar uses the exact count; for others it uses the upper end of the estimate, so it errs on the full side. You can also add a custom limit.
Estimating Cost
API prices are quoted per million tokens and change often, which is why the tool deliberately does not show them. The calculation itself is simple:
cost = (input tokens ÷ 1,000,000 × input price)
+ (output tokens ÷ 1,000,000 × output price)Input, cached input and output tokens can each have a different price, and output is usually the more expensive side. Take current prices from your provider’s pricing page, and count your real text rather than estimating from words.
Three Everyday Token Questions, Worked Through
Will my long report fit in one prompt?
A 30-page single-spaced report in 12-point Times New Roman holds about 15,000 words (500 words per page). At the 1.25 tokens per word we measured on ordinary prose, that is roughly 18,750 tokens, plus whatever instructions you add. An estimate like this tells you whether you are anywhere near a limit; to be sure, paste the real text into the counter, because tables, numbers and references push the ratio up.
Is it worth shortening my instructions?
If the same instructions are sent with every request, yes. A system prompt that is 400 tokens longer than it needs to be costs those 400 tokens on every call, and the waste multiplies with traffic. Count it, cut repetition and examples that do not change the output, and count again.
Why does a character counter disagree with the token counter?
They measure different things for different limits. Social networks and form fields limit characters, which the Character Counter measures. Language models limit tokens. At the 4.8 characters per token we measured on plain prose, a 280-character post is around 60 tokens; one packed with numbers, links and emoji can be far more.
How to Count Tokens With the AI Token Counter
- Paste your prompt or open a file. Text files such as .txt, .md, .json, .csv and source code up to 10 MB can be opened. The tokenizer (about 2.4 MB) loads once from this site the first time.
- Read the exact count. The large number is the o200k_base count, with characters, words and characters per token beside it.
- Check the context table for the models you use, or set a custom limit.
- Inspect the tokens. The visualizer colors each token. Turn on “Show whitespace” to see spaces as · and line breaks as ↵, or “Show token IDs” to see each token’s number. Long texts show the first 3,000 tokens. “Copy token IDs” copies the full list as a JSON array.
Everything runs in your browser. Your text is never uploaded, and strings that look like special tokens, such as <|endoftext|>, are counted as ordinary text (7 tokens) rather than causing an error.
Six Ways to Use Fewer Tokens
- Compact your JSON. A small object pretty-printed with two-space indentation took 37 tokens; the same object on one line took 20. The JSON Formatter & Validator can minify it, and our JSON formatting guide explains when each form fits.
- Remove repeated context. Instructions repeated in every message, or a document pasted twice, cost tokens every time.
- Clean the whitespace. Runs of spaces and blank lines from copied text add up; Remove Extra Spaces tidies them.
- Summarize long background. If the model only needs the gist of a long document, a summary is far cheaper than the full text. The Text Summarizer can help.
- Drop decorative formatting. Markdown tables, long separator lines and emoji all tokenize poorly.
- Measure, then trim. Count the real text, cut the least useful parts, and count again. Word counts from the Word Counter are useful for writing targets, but token counts are what models enforce.