Why does AI charge by tokens? How many tokens is one Chinese character?
8 min readWhen we chat with AI, it doesn't read character by character like humans do. Nor does it understand text word by word. The fundamental reading unit of AI is called a Token -- a subword fragment that sits somewhere between a character and a full word.
A Token could be a complete English word (like the), part of a word (like un + happi + ness), or a single Chinese character or a few bytes. How exactly text gets split depends on the model's Tokenizer.
Type any text below and watch how it gets split into Tokens:
Tokens are like LEGO bricks -- AI doesn't understand complete sentences; it breaks text into small "bricks" and processes them one by one.
Most modern large language models use the Byte Pair Encoding (BPE) algorithm to build their tokenizers. The core idea of BPE is remarkably simple: repeatedly merge the most frequently occurring adjacent symbol pairs.
The BPE training process works as follows:
1. Split all text into the smallest units (characters or bytes)
2. Count the frequency of all adjacent symbol pairs
3. Merge the most frequent pair into a new symbol
4. Repeat steps 2-3 until the vocabulary reaches the desired size
Watch how the word lowest gets progressively merged via BPE (based on frequency statistics):
BPE is like finding shorthand patterns -- if you notice "th" always appears together, you combine it into a single new symbol.
Token efficiency varies greatly across languages. English words are made up of 26 letters, and common words (like the, is) are often a single Token. Chinese, on the other hand, has thousands of commonly used characters, each carrying far more information than a single English letter -- but the tokenizer needs more Tokens to encode them.
In short: expressing the same meaning in Chinese often requires more Tokens, even though the character count may be lower.
Chinese is like a compressed archive -- high information density, but AI needs more "decoding steps" to process it.
The cost of calling an AI model is directly tied to the number of Tokens. Whether it's the Prompt (input) sent to the model or the Completion (output) generated by the model, both are billed per Token. Output is typically more expensive than input because generating text requires more computation.
Below is a pricing reference for major models per 1 million Tokens (USD):
| Model | Input Price | Output Price | Cached Input |
|---|---|---|---|
| GPT-5.4 | $2.50 | $15.00 | $1.25 |
| Claude Opus 4.6 | $5.00 | $25.00 | $0.50 |
| Claude Sonnet 4.6 | $3.00 | $15.00 | $0.30 |
| Gemini 2.5 Pro | $1.25 | $10.00 | - |
| DeepSeek V3.2 | $0.28 | $0.42 | $0.028 |
| Qwen3.5 Plus | $0.26 | $1.56 | - |
* Prices are public pricing as of March 2026 (USD / 1M tokens) and may change over time. Cached Input refers to Prompt Caching, which can significantly reduce costs for repeated context. On-device inference (e.g., AtomGradient Runtime) has zero Token costs.
As you can see, there are huge price differences between models. DeepSeek V3.2 and Qwen3.5 offer extremely competitive pricing (output at just $0.42-1.56/M), while Claude Opus 4.6 and GPT-5.4 each have their own strengths in capability. When choosing a model, you need to balance cost against capability.
Tokens are AI's "word count" -- just like phone plans charge by the minute, AI charges by the Token.
AI doesn't read by character or by word. It processes text using subword units (Tokens) produced by a tokenizer.
Byte Pair Encoding starts from the smallest units and repeatedly merges high-frequency adjacent pairs to build an efficient vocabulary.
Chinese has high information density but higher Token overhead. The same meaning in Chinese typically consumes more Tokens than in English.
Understanding Tokens helps you estimate AI usage costs and choose the right model and optimization strategy.
Understand Tokens, and you understand AI's "unit of language" and "unit of billing."