You do not need to understand how a large language model works internally to make good decisions about using one. But you do need an accurate mental model, because the wrong one leads directly to the wrong expectations — and most of the disappointment we see traces back to that.
What it actually does
A large language model predicts what text should come next. That is the whole mechanism. Given everything so far, it produces a likely continuation, one piece at a time, each new piece informed by everything before it.
It learned to do this by processing an enormous quantity of text and repeatedly adjusting itself to predict better. Nothing was stored as facts in a database. What was built up is a statistical sense of how language fits together — which, because language encodes knowledge, ends up carrying a great deal of knowledge with it.
Why this explains so much
Once you hold that model, the behaviour stops being surprising.
Why it writes fluently but sometimes states false things. Fluency and truth are different properties. The system optimises for plausible continuation. A plausible-sounding wrong answer and a plausible-sounding right answer look similar from the inside.
Why it is confident when wrong. There is no separate module assessing certainty. The hedging language in an answer is itself predicted text, not a readout of internal confidence.
Why phrasing changes the output so much. You are changing the context that the prediction is conditioned on. Different context, different likely continuation.
Why it is bad at arithmetic. Calculation is not prediction. It can often produce the right answer for common sums because it has seen them, and fail on unusual ones for the same reason. This is why serious systems hand arithmetic to a calculator rather than the model.
Why it does not know about last week. Training happened at a point in time. Anything after that is absent unless supplied in the conversation.
What it is genuinely good at
Transformation of text is the core strength: summarising, rewriting for a different audience, changing format, translating, extracting structure from prose, classifying, and drafting from a description.
Notice the pattern — these are tasks where the necessary information is present in the input and the work is reshaping it. That is exactly where prediction over language performs well.
What it is not
It is not a database. It has no reliable index of facts and cannot tell you what it does not know.
It is not a reasoning engine in the way a rules system is. It produces text that resembles reasoning, which is often useful and occasionally arrives at a confidently wrong conclusion by a route that reads convincingly.
It is not deterministic by default. The same question can produce different answers, which matters if you were expecting reproducibility.
The practical consequence
Design around the strength and guard the weakness. Supply the relevant information rather than relying on what the model absorbed during training. Use it to transform and draft rather than to be the authority. And put a check between its output and anything that matters.
Companies that internalise this build things that work. Companies that treat it as an oracle build things that work in the demo.
