Someone uploads a ninety-page contract, asks a question about the final section, and gets an answer that ignores it entirely — or an error. This is the point at which most people first meet tokens and context windows, and it is worth understanding before it happens.
Tokens
Language models do not read letters or words. Text is broken into tokens: common words are usually a single token, longer or unusual words split into several, and punctuation and spaces count too.
A workable rule of thumb for English is that a token averages about three-quarters of a word. German tends to be less efficient because compound nouns split into pieces, so the same text in German typically costs more tokens than in English. This matters for both cost and capacity.
The context window
The context window is the total amount of text the model can hold in view at once. It covers everything: your instructions, any documents you supplied, the conversation so far, and the answer being generated.
It is a hard limit, not a preference. Exceeding it means something is dropped or the request is rejected — and which of those happens depends on the system, which is why the same mistake shows up sometimes as an error and sometimes as an answer that quietly ignores half your document.
Why long conversations degrade
Every exchange adds to the accumulated context. In a long conversation, early messages eventually fall out of the window. The system does not announce this. It simply stops knowing what you agreed at the start, which reads as it "forgetting" or becoming less consistent.
The practical response is to start a fresh conversation for a new task, and to restate important constraints rather than assuming they are still in view.
Bigger windows are not the whole answer
Modern models offer large windows, and it is tempting to conclude the problem is solved. Two reasons it is not.
First, cost scales with what you send. Putting an entire document library into every request is expensive and repetitive.
Second, and more important: performance is not uniform across a very long context. Material in the middle of a large body of text tends to be used less reliably than material at the beginning or end. Filling the window is not the same as the model attending to all of it equally.
The better approach
Rather than supplying everything and hoping, find the relevant parts first and supply those. This is what retrieval-based systems do: documents are split into sections, indexed, and the sections relevant to the question are fetched and included.
The result is cheaper, more accurate, and scales past any window size — because you are sending five relevant pages instead of nine hundred mostly irrelevant ones.
What this means in practice
If you are evaluating a system that will handle long documents, ask how it deals with material exceeding the window. "We use a model with a large context" is a weaker answer than a description of how relevant sections are located.
And if you are using a chat assistant for a substantial task, keep conversations focused, restate what matters, and be suspicious of an assistant that seems to have quietly forgotten the brief.
