Context window limits: why a longer conversation can make an agent less reliable, not more informed

A model’s context window — the amount of text it can hold in a single conversation, including a long document, a chat history or a large set of retrieved passages — keeps growing, and it is tempting to treat that growth as a straightforward upgrade: more room means more information the agent has access to. What research on how models actually use that space shows is that access is not the same thing as reliable use, and stuffing a context window full does not guarantee the model weighs every part of it equally.

What "lost in the middle" means

A team from Stanford and other institutions, in Lost in the Middle: How Language Models Use Long Contexts, published in the Transactions of the Association for Computational Linguistics (2023), tested how well models retrieve relevant information depending on where it sits in a long input. The finding: performance is strongest when the relevant information is near the beginning or the end of the context, and drops noticeably when it sits in the middle — and this degradation gets worse, not better, as the context grows longer, even in models explicitly built for long-context use.

What this means for an agent working over long documents

For an agent asked to answer from a long contract, a large ticket history or a lengthy report, the practical consequence is that dumping the entire document into context and trusting the model to find the right passage is a weaker design than retrieving the specific relevant section first and feeding the model a shorter, targeted input. A bigger context window makes more information reachable — it does not, on its own, make an agent better at finding the right piece of it.

• →Retrieve the relevant passage before generating an answer, rather than relying on the model to locate it inside a very long input.
• →Put the most critical information near the start or end of what is sent to the model when a long context is unavoidable.
• →Test an agent specifically on long-document tasks before assuming a larger context window has solved an accuracy problem — it often has not.