The mistake is intuitive. You have a context window that can hold hundreds of thousands of tokens. You have information the model needs. You put the information in the context. The model reads it. What is the problem?
The problem is that "reads" is doing a lot of work in that sentence. A language model does not read context the way a human reads a document - linearly, building up a model of the content, updating understanding as new information arrives. It processes the entire context at once through attention mechanisms that weight different parts of the input differently depending on the query. The implication is that context position matters enormously. Information near the end of a long context tends to get more weight than information buried in the middle. This is not a bug that will get patched; it is structural to how attention works.
There are good empirical demonstrations of this. Ask a model to answer a question whose answer appears in the first third of a very long document, then in the last third, and you will often get different quality responses. The model is not equally attentive across the context. Researchers call this the lost-in-the-middle problem, which is a good name for it, because that is exactly what happens - relevant information, placed in the middle of a long context, gets underweighted relative to information at the boundaries.
What does this mean practically? A few things. First, the order in which you put information into a context window is not neutral. If the most important information is your system prompt and it is buried under three thousand tokens of retrieved documents before the user query, you may be hurting yourself. Put the highest-priority instructions close to the query. Second, stuffing the context with marginally relevant documents in hopes that the model will figure out which parts matter is not a good strategy. Selectivity before context injection consistently outperforms completeness. Retrieve better, not more. Third, very long contexts are not a substitute for structured memory. If your application needs to maintain state across a long interaction, build a real memory system - summarize, extract, and store relevant information explicitly rather than dragging the entire conversation history along in every call.
The deeper issue is that treating the context window as memory encourages a kind of laziness that eventually bites you. It is easy to throw more tokens at a problem; it is harder to think carefully about what information actually matters for a given query and how to surface it effectively. The teams that do the harder thing consistently outperform the teams that do not, even when they have access to identical models and similar compute budgets.
Context windows will keep getting larger. That is a useful capability expansion - there are genuine use cases for very long contexts. But a larger window does not fix the attention dynamics. It just gives you a larger space in which to make the same mistakes.