The idea
At each step the model assigns a probability to every possible next token, then samples one, usually with some randomness. Temperature controls how much: low means more predictable, high means more varied and more likely to drift.
Because each token builds on the last, a small difference early on can compound into a very different answer.
Mental model
You're not designing an output. You're designing a distribution of outputs: typical ones, great ones, and bad ones that show up some percentage of the time.
Why it matters when you build
- One good demo tells you almost nothing. Run the same input many times, across many test cases. That's the seed of an eval.
- Design for the bad tail. Give users regenerate, edit, undo and "report a problem." Assume some answers will miss.
- Pick temperature per task. Use low for extraction, classification and data. Use higher for brainstorming and creative copy.
- Constrain when you need consistency. Structured output, examples and tighter instructions narrow the distribution.
Gotchas
- Even temperature 0 isn't perfectly deterministic on most providers. Don't build features that depend on byte-identical output.