Language models & chatbots
A token is the unit of text a language model reads and writes; it is usually a word, part of a word, or a punctuation mark, roughly four characters in English.
Language models do not see letters or whole sentences. Before text goes in, a tokenizer chops it into pieces from a fixed vocabulary of tens of thousands of chunks. Common words are one token. Rare or long words are split into several. Spaces and punctuation get their own tokens too.
The model then predicts one token at a time. Everything about an LLM is measured in tokens: how long a prompt can be, how much you pay when using an API, and how fast it responds. A rough rule for English is that 1,000 tokens is about 750 words.
Tokens explain some odd behavior. A model may struggle to count the letters in a word because it never sees the letters, only the chunks. Other languages often need more tokens per sentence than English, making them slower and costlier to process.
The word 'unbelievable' might become three tokens, 'un,' 'believ,' and 'able,' while a common word like 'the' is a single token.