Next-token prediction
Every essay, poem and line of code a model has ever written came from the same tiny step, repeated. Understand that one step and most of a model's strange behaviour stops being strange.
In this lesson you'll learn to
- Describe how a model writes a reply one token at a time, feeding each choice back in.
- Explain how logits become probabilities, and what those probabilities mean.
- Use next-token prediction to explain confident mistakes, sensitivity to wording, and why a model can't take back a word.
- Explain why writing reasoning steps first can improve answers, and why output speed is measured per token.
Create a free account to open this lesson
It's free. You'll get every free lesson, quizzes that save your progress, daily flashcards and streaks.