Tokens and tokenization
A model that can write a sonnet once struggled to count the r's in "strawberry". It isn't careless. It has never seen the letters.
In this lesson you'll learn to
- Explain what a token is, and how text becomes a list of token ids before a model sees it.
- Explain how tokens set cost, length limits and speed, and why they cause spelling and counting mistakes.
- Explain why the same message can take more tokens in some languages, and what that does to cost.
- Describe how a tokenizer's vocabulary is built with byte-pair encoding, and count tokens in code.
Create a free account to open this lesson
It's free. You'll get every free lesson, quizzes that save your progress, daily flashcards and streaks.