Skip to content

What Are AI Tokens? How Text Becomes Tokens (With Real Counts)

The Short Answer

A token is a chunk of text that a language model reads as one unit: often a whole common word together with the space before it, sometimes part of a longer word, a punctuation mark, a group of digits or a piece of an emoji. Models read, limit and bill text in tokens, not in words or characters.

For ordinary English, OpenAI’s rule of thumb is that one token is about four characters or three-quarters of a word, so 100 tokens are roughly 75 words (OpenAI Help Center, “Understanding and counting tokens”, checked on 2026-10-14). The same article stresses that these are estimates: the exact number depends on the model’s encoding, the language and the kind of text. The examples below show how far real text can drift from the rule, and how to get an exact count with WritePadPro’s AI Token Counter.

How Text Becomes Tokens

OpenAI’s current models, including GPT-4o, GPT-4.1, GPT-5 and the o-series, use a byte-pair encoding called o200k_base, with a vocabulary of about 200,000 tokens. Turning text into tokens takes three steps:

  1. Pre-splitting. A pattern separates the text into words, numbers, punctuation and whitespace. A word usually keeps the space in front of it, which is why “ world” with its leading space is a single token.
  2. Bytes. Each piece is converted into UTF-8 bytes. An English letter is one byte; an accented letter, a Hindi character or an emoji takes several.
  3. Merging. Starting from single bytes, the most frequent adjacent pairs are merged step by step, following a fixed ranking learned from training data, until no ranked merge is left. Each remaining chunk is one token with a number (its ID) in the vocabulary.

Common words were merged into single tokens during training. Rare words, names, long numbers and most non-English text were not, so they break into several pieces.

Real Token Counts You Can Check

Every count below was produced with the o200k_base encoding, the same open-source tokenizer the AI Token Counter runs in your browser. The vertical bars show where the token boundaries fall.

TextTokensHow it splits
hello world2hello | world
Hello, world!4Hello | , | world | !
The quick brown fox jumps over the lazy dog.10one token per word, plus the period
tokenization2token | ization
unbelievably3un | bel | ievably
WritePadPro3Write | Pad | Pro
antidisestablishmentarianism6ant | idis | est | ablishment | arian | ism
12345673123 | 456 | 7
20262202 | 6
$1,299.996$ | 1 | , | 299 | . | 99
2026-11-066202 | 6 | - | 11 | - | 06
https://www.writepadpro.com/tools/ai-token-counter12https | :// | www | .write | pad | pro | .com | /tools | / | ai | -token | -counter
👍1one token
👍🏽 (with a skin tone)3the thumb, then two pieces for the skin-tone bytes
🇺🇸 (flag)4two regional-indicator symbols, two pieces each
four spaces1whitespace runs are often one token

Two patterns stand out. Numbers are cut into groups of up to three digits, so long IDs, prices and dates cost far more tokens than their length suggests. And anything unusual, such as brand names, rare words, URLs and combined emoji, splits into pieces.

Words vs. Tokens on Real Text

Short examples exaggerate the differences, so we also measured longer text. The body paragraphs of our guide on how to paraphrase without plagiarizing contain 1,029 words of ordinary English prose. In o200k_base they come to 1,284 tokens: 1.25 tokens per word, 4.8 characters per token, or about 800 words per 1,000 tokens. That is a little better than the 750-words rule of thumb, which is fine for planning but not for hard limits.

Code is a different story. One of the JavaScript files behind this site, 12,553 characters long, came to 3,453 tokens, or 3.6 characters per token; a 15,530-character stylesheet came to 4,536 tokens (3.4 characters per token). Code is full of symbols, short identifiers and indentation, and all of them cost tokens.

Language matters even more. Here is the same greeting in seven languages:

LanguageTextCharactersTokens
EnglishGood morning, how are you?267
SpanishBuenos días, ¿cómo estás?257
GermanGuten Morgen, wie geht es dir?309
RussianДоброе утро, как дела?229
Hindiसुप्रभात, आप कैसे हैं?2210
Japaneseおはようございます、お元気ですか?1710
Chinese早上好,你好吗?86

A single sentence is a tiny sample, so treat these as an illustration rather than a ratio. The pattern is consistent with how tokenizers are built: vocabularies learned mostly from English and code turn common English words into single tokens, while other scripts are split into more pieces per word. Characters per token is meaningless for Chinese and Japanese, where one character can carry a whole word.

Why Claude, Gemini and Llama Count Differently

Every model family has its own tokenizer, so the same text gives a different count in each. Exact counts for Claude and Gemini come from the providers’ own token-counting APIs, which means sending them your text. The AI Token Counter does not upload anything, so for those models it shows a clearly labeled estimate range based on characters per token:

Model familyEstimate the tool uses
OpenAI (GPT-4o, GPT-4.1, GPT-5, o-series)Exact count with o200k_base
Claudecharacters ÷ 4 to characters ÷ 3
Geminicharacters ÷ 4.5 to characters ÷ 3.5
Llama 3 and latercharacters ÷ 4.5 to characters ÷ 3.5
Mistralcharacters ÷ 4 to characters ÷ 3.2

Chinese, Japanese and Korean characters are counted separately at 0.8 to 1.5 tokens each, because the characters-per-token rule does not apply to them. The ranges suit ordinary prose; code, tables and long numbers can fall outside them. For billing or hard limits, confirm with the provider.

Context Windows: What Has to Fit

A model’s context window is the maximum number of tokens it can handle in one request. For most models it covers everything at once: the system prompt, the conversation so far, any documents you paste, and the reply. Reasoning models also produce internal reasoning tokens, which OpenAI’s help article notes are billed as output tokens even though you never see them.

Three practical consequences:

  • Leave room for the answer. A prompt that fills nearly the whole window leaves almost nothing for the reply.
  • Chats grow. Every turn adds the previous messages again, so a long conversation can reach the limit even with short prompts.
  • Plain-text counts are a floor. Message formatting, tool definitions, images and files add tokens that a text count does not include.

The AI Token Counter’s table shows how much of each popular model’s window your text would use. For OpenAI models the bar uses the exact count; for others it uses the upper end of the estimate, so it errs on the full side. You can also add a custom limit.

Estimating Cost

API prices are quoted per million tokens and change often, which is why the tool deliberately does not show them. The calculation itself is simple:

cost = (input tokens ÷ 1,000,000 × input price)
     + (output tokens ÷ 1,000,000 × output price)

Input, cached input and output tokens can each have a different price, and output is usually the more expensive side. Take current prices from your provider’s pricing page, and count your real text rather than estimating from words.

Three Everyday Token Questions, Worked Through

Will my long report fit in one prompt?

A 30-page single-spaced report in 12-point Times New Roman holds about 15,000 words (500 words per page). At the 1.25 tokens per word we measured on ordinary prose, that is roughly 18,750 tokens, plus whatever instructions you add. An estimate like this tells you whether you are anywhere near a limit; to be sure, paste the real text into the counter, because tables, numbers and references push the ratio up.

Is it worth shortening my instructions?

If the same instructions are sent with every request, yes. A system prompt that is 400 tokens longer than it needs to be costs those 400 tokens on every call, and the waste multiplies with traffic. Count it, cut repetition and examples that do not change the output, and count again.

Why does a character counter disagree with the token counter?

They measure different things for different limits. Social networks and form fields limit characters, which the Character Counter measures. Language models limit tokens. At the 4.8 characters per token we measured on plain prose, a 280-character post is around 60 tokens; one packed with numbers, links and emoji can be far more.

How to Count Tokens With the AI Token Counter

  1. Paste your prompt or open a file. Text files such as .txt, .md, .json, .csv and source code up to 10 MB can be opened. The tokenizer (about 2.4 MB) loads once from this site the first time.
  2. Read the exact count. The large number is the o200k_base count, with characters, words and characters per token beside it.
  3. Check the context table for the models you use, or set a custom limit.
  4. Inspect the tokens. The visualizer colors each token. Turn on “Show whitespace” to see spaces as · and line breaks as ↵, or “Show token IDs” to see each token’s number. Long texts show the first 3,000 tokens. “Copy token IDs” copies the full list as a JSON array.

Everything runs in your browser. Your text is never uploaded, and strings that look like special tokens, such as <|endoftext|>, are counted as ordinary text (7 tokens) rather than causing an error.

Six Ways to Use Fewer Tokens

  • Compact your JSON. A small object pretty-printed with two-space indentation took 37 tokens; the same object on one line took 20. The JSON Formatter & Validator can minify it, and our JSON formatting guide explains when each form fits.
  • Remove repeated context. Instructions repeated in every message, or a document pasted twice, cost tokens every time.
  • Clean the whitespace. Runs of spaces and blank lines from copied text add up; Remove Extra Spaces tidies them.
  • Summarize long background. If the model only needs the gist of a long document, a summary is far cheaper than the full text. The Text Summarizer can help.
  • Drop decorative formatting. Markdown tables, long separator lines and emoji all tokenize poorly.
  • Measure, then trim. Count the real text, cut the least useful parts, and count again. Word counts from the Word Counter are useful for writing targets, but token counts are what models enforce.

Frequently Asked Questions

How many words is 1,000 tokens?

For ordinary English prose, roughly 750 to 800 words. OpenAI’s rule of thumb gives about 750, and the body paragraphs of one of our own guides came to about 800 words per 1,000 tokens. Code, numbers, URLs and most other languages give far fewer words per 1,000 tokens.

Why does a number like 1234567 use three tokens?

The o200k_base tokenizer splits digit runs into groups of up to three digits, so 1234567 becomes 123, 456 and 7. Long IDs, phone numbers, prices and dates therefore use more tokens than their character count suggests.

Is the AI Token Counter exact for ChatGPT models?

Yes for the text itself. It runs the o200k_base encoding used by GPT-4o, GPT-4.1, GPT-5 and the o-series models. A full chat request adds a few formatting tokens per message, and images, files and tool definitions are counted separately by the API.

Why are counts for Claude and Gemini only estimates?

Each model family uses its own tokenizer, and exact counts come from the providers’ token-counting APIs, which require sending them your text. The tool shows a clearly labeled range based on characters per token instead of uploading anything.

Do spaces and line breaks count as tokens?

Yes. A space usually attaches to the word after it, so it costs nothing extra, but runs of spaces, tabs, indentation and blank lines become tokens of their own. Turn on Show whitespace in the visualizer to see them.

Why do non-English texts use more tokens?

Tokenizer vocabularies are learned mostly from English and code, so common English words are single tokens while words in many other languages split into several pieces. In our seven-language example, the same greeting took 7 tokens in English and 10 in Hindi and in Japanese.

Try These Tools

Related Articles