What are tokens in AI?
Tokens are text pieces models operate on — sometimes a whole word, sometimes part of a word or a symbol. Before “reasoning”, the model converts your words into a list of token IDs.
Token count affects cost, speed, and how much context fits in memory (context window). Tokenization is the first real step in every LLM pipeline.
Try live tokenization in the mej7.com lab — type text and see how it splits.