How Large Language Models Actually Generate Text
AI Tools - 3 min read - Published 2026-09-06 - Updated 2026-10-19
Large language models (LLMs) power chat assistants, writing tools, summarizers and translators. Their answers can feel like conversation with someone who knows everything, but the underlying mechanism is simpler and more mechanical than that. Understanding it explains both why they're so useful and why they make the particular kinds of mistakes they do.
Text as tokens
Models don't read letters or whole words. Text is split into tokens: common words are often a single token, while rarer words are split into pieces, and spaces and punctuation count too. In English, a token averages roughly three-quarters of a word, so 1,000 tokens is around 750 words. Other languages often need more tokens for the same meaning.
Tokens explain some odd behavior. Because a model sees chunks rather than individual letters, tasks such as counting the letters in a word or reversing a string can trip it up, even when it handles much harder questions well.
One token at a time
At its core, an LLM does one thing: given the text so far, it calculates a probability for every possible next token. After "The capital of France is", the token " Paris" gets a very high probability. The system picks a token, adds it to the text, and repeats - so a long answer is built one token at a time, each choice based on everything before it, including your question and any instructions.
The model learns these probabilities in training, by processing enormous amounts of text and adjusting billions of internal numbers so that its predictions match what actually came next. That's how it picks up grammar, facts, styles of writing and patterns of reasoning: all of them help predict text. Most assistants are then further trained with human feedback to follow instructions, be helpful and decline harmful requests.
Why answers vary
If the system always chose the single most likely token, answers would be repetitive and dull. Instead it usually samples: it picks randomly, weighted by the probabilities. A setting called temperature controls how adventurous that sampling is. Low temperature sticks closely to the most likely tokens, giving consistent, focused answers; higher temperature allows less likely choices, giving more varied and creative text.
That's why asking the same question twice can give different wording, or occasionally a different answer. For tasks where consistency matters, such as extracting data, lower temperatures and clear instructions help.
Why models state false things confidently
A model produces text that is plausible given its training, which isn't the same as checking that it's true. When it lacks the right information, it can still produce a fluent, confident answer - an invented statistic, a book that doesn't exist, a citation with a real-looking but fake title. This is often called hallucination.
Models also have a training cutoff: on their own they don't know about events after their training data was collected, unless the tool gives them access to search or documents. Some assistants do, and then their answers can draw on current sources, but it's worth checking which kind you're using.
The context window
Everything the model considers at once - your instructions, the conversation so far, any pasted document and its own reply - must fit in its context window, measured in tokens. Modern models have large windows, but in very long conversations earlier details can still be dropped or given less attention. For long documents, asking about one section at a time, or restating key requirements, often works better.
Using them well
Give context and be specific: who the text is for, the format you want, the length, and any facts it must use. Paste the source material when you want an answer based on it, such as a document to summarize, rather than relying on what the model remembers. Ask it to say when it isn't sure.
Check anything that matters: facts, figures, quotes, code and legal, medical or financial claims. LLMs are excellent at drafting, rewording, summarizing, brainstorming and explaining, and much less reliable as a source of truth. Don't paste passwords, confidential business data or other people's personal information into an AI tool unless you know how the service handles it.