Tokens
Your words become tokens
Neural networks don't process words as human concepts. Text is first converted into <b>tokens</b> — pieces of text represented numerically.
Token IDs depend entirely on a model's specific tokenizer/vocabulary — the IDs shown here are illustrative, not universal.
Long or rare words are often split into smaller pieces. For example, "unbelievable" might become:
This is just an illustrative example — real splitting depends on the specific tokenizer in use.
In more detail
A language model cannot read letters or words directly. It works with numbers, so the first step is to cut the text into pieces called tokens and to replace each piece with an integer ID taken from a fixed vocabulary.
A token may be a whole word, a part of a word, a single character, a punctuation mark, or a word together with its leading space. Most modern models use subword tokenization, for example byte-pair encoding (BPE). The vocabulary is built by starting from small units and repeatedly merging the pairs that occur most often in a large sample of text. Frequent words usually become a single token; rare or long words are split into several pieces. This keeps the vocabulary at a manageable size (typically tens of thousands to a few hundred thousand entries) while still allowing any text to be represented, often by falling back to individual bytes.
Why this matters in practice:
- The context window and, for many services, the price are counted in tokens, not words. In English, one token is on average a bit less than one word (a common rule of thumb is about three quarters of a word), but the ratio varies. Languages that are less represented in a tokenizer's training text, such as Albanian, often need more tokens per word.
- Numbers, code and unusual spellings can split in unexpected ways. This is one reason why models can struggle with tasks such as counting the letters in a word.
- Token IDs belong to one tokenizer. The same text receives different IDs in different models, and an ID has no meaning on its own. The next step turns it into a vector.
Simplification: the splits and IDs in the visualization are illustrative. Real tokenizers use their own vocabularies and rules.