Transformers and attention
In "The cat didn't eat because it was full", you know instantly that "it" means the cat. Getting a machine to make that link, at scale, is the idea that made today's models possible.
In this lesson you'll learn to
- Explain how tokens become embeddings, lists of numbers where similar meanings sit close together.
- Explain attention as each token deciding which earlier tokens matter to it.
- Describe how stacked layers refine the meaning and end in next-token scores, and why parameters are counted in billions.
- Explain why attention gets more expensive as the text grows, and what the KV cache saves.
Create a free account to open this lesson
It's free. You'll get every free lesson, quizzes that save your progress, daily flashcards and streaks.