Tokens & Tokenization

Why does AI charge by tokens? How many tokens is one Chinese character?

NLP Tokenization BPE
8 min read
Chapter 01

What is a Token?

When we chat with AI, it doesn't read character by character like humans do. Nor does it understand text word by word. The fundamental reading unit of AI is called a Token -- a subword fragment that sits somewhere between a character and a full word.

A Token could be a complete English word (like the), part of a word (like un + happi + ness), or a single Chinese character or a few bytes. How exactly text gets split depends on the model's Tokenizer.

// Interactive Demo: Live Tokenization

Type any text below and watch how it gets split into Tokens:

Tokens are like LEGO bricks -- AI doesn't understand complete sentences; it breaks text into small "bricks" and processes them one by one.

Chapter 02

BPE Algorithm

Most modern large language models use the Byte Pair Encoding (BPE) algorithm to build their tokenizers. The core idea of BPE is remarkably simple: repeatedly merge the most frequently occurring adjacent symbol pairs.

The BPE training process works as follows:

1. Split all text into the smallest units (characters or bytes)
2. Count the frequency of all adjacent symbol pairs
3. Merge the most frequent pair into a new symbol
4. Repeat steps 2-3 until the vocabulary reaches the desired size

// Interactive Demo: BPE Merge Process

Watch how the word lowest gets progressively merged via BPE (based on frequency statistics):

BPE is like finding shorthand patterns -- if you notice "th" always appears together, you combine it into a single new symbol.

Chapter 03

Chinese vs English

Token efficiency varies greatly across languages. English words are made up of 26 letters, and common words (like the, is) are often a single Token. Chinese, on the other hand, has thousands of commonly used characters, each carrying far more information than a single English letter -- but the tokenizer needs more Tokens to encode them.

In short: expressing the same meaning in Chinese often requires more Tokens, even though the character count may be lower.

Example 1

English
The cat sat on the mat
6
Tokens
Chinese
猫坐在垫子上
6
Tokens

Example 2

English
Artificial intelligence is transforming our daily lives
7
Tokens
Chinese
人工智能正在改变我们的日常生活
11
Tokens

Chinese is like a compressed archive -- high information density, but AI needs more "decoding steps" to process it.

Chapter 04

Tokens & Pricing

The cost of calling an AI model is directly tied to the number of Tokens. Whether it's the Prompt (input) sent to the model or the Completion (output) generated by the model, both are billed per Token. Output is typically more expensive than input because generating text requires more computation.

Below is a pricing reference for major models per 1 million Tokens (USD):

Model Input Price Output Price Cached Input
GPT-5.4 $2.50 $15.00 $1.25
Claude Opus 4.6 $5.00 $25.00 $0.50
Claude Sonnet 4.6 $3.00 $15.00 $0.30
Gemini 2.5 Pro $1.25 $10.00 -
DeepSeek V3.2 $0.28 $0.42 $0.028
Qwen3.5 Plus $0.26 $1.56 -

* Prices are public pricing as of March 2026 (USD / 1M tokens) and may change over time. Cached Input refers to Prompt Caching, which can significantly reduce costs for repeated context. On-device inference (e.g., AtomGradient Runtime) has zero Token costs.

As you can see, there are huge price differences between models. DeepSeek V3.2 and Qwen3.5 offer extremely competitive pricing (output at just $0.42-1.56/M), while Claude Opus 4.6 and GPT-5.4 each have their own strengths in capability. When choosing a model, you need to balance cost against capability.

Tokens are AI's "word count" -- just like phone plans charge by the minute, AI charges by the Token.

Chapter 05

Summary

🧩

Tokens are Subword Fragments

AI doesn't read by character or by word. It processes text using subword units (Tokens) produced by a tokenizer.

🔗

BPE Merges Iteratively

Byte Pair Encoding starts from the smallest units and repeatedly merges high-frequency adjacent pairs to build an efficient vocabulary.

🌏

Language Affects Efficiency

Chinese has high information density but higher Token overhead. The same meaning in Chinese typically consumes more Tokens than in English.

💰

Token = Billing Unit

Understanding Tokens helps you estimate AI usage costs and choose the right model and optimization strategy.

Understand Tokens, and you understand AI's "unit of language" and "unit of billing."

Next: Transformer Attention →