Tokenization

Tokenization is the process of breaking text into smaller pieces called tokens — words, parts of words, or characters — so a language model can process it.

Data & AI

Models don't read raw text; they read sequences of tokens that have been converted to numbers. Tokens are often sub-word chunks, so 'unhappiness' might split into 'un', 'happi', and 'ness'. This lets models handle rare or new words by combining familiar pieces.

Tokenization is the first step in nearly every NLP pipeline. The number of tokens also drives cost and length limits in large language models, since they charge and reason per token rather than per word.

Examples

Python

text = "Learning AI is fun!"
tokens = text.replace("!", " !").split()
print(tokens)
print("token count:", len(tokens))

Output:

['Learning', 'AI', 'is', 'fun', '!']
token count: 5

Learn it in a lesson

See also

Back to the Dictionary