When a bigger context window isn't the answer
Long-context models tempt you to skip retrieval. Why that gets expensive and inaccurate, and what to do instead.
Every jump in context length brings the same suggestion: drop the retrieval layer and put everything in the prompt. It is a reasonable instinct and usually the wrong move.
You pay for every token, every time
Stuffing a hundred thousand tokens of context into each request multiplies your bill by the number of requests. Retrieval exists partly to make the economics work, and that reason does not go away when the window grows.
Why a bigger window is not more signal
Attention is uneven across a long context — material in the middle is reliably weaker.
- Start of context5
Strongest recall
- Middle of context2
Where information goes to be forgotten
- End of context5
Strong again — recency helps
Attention is not uniform
Models attend unevenly across a long context, and material in the middle is reliably weaker than material at the edges. A precise five-thousand-token context routinely beats a sloppy hundred-thousand-token one — more input is not more signal.
Retrieve, or fill the window?
- Answer lives in a few known placesRetrieve
- Whole document matters, can't predict whereLong context
- High request volume, cost-sensitiveRetrieve
- Long conversation, early turns still relevantLong context
- Best of bothRetrieve generously, let the window absorb it
Where long context genuinely wins
There are real cases for it, and they share a shape: the whole document matters and you cannot know in advance which part.
- Whole-document reasoning — summarise this contract, find inconsistencies across it
- Long multi-turn conversations where earlier turns stay relevant
- Codebase-wide questions that span many files
- Cutting a retrieval step for a prototype, deliberately and temporarily
The pattern that usually wins
Retrieve generously, then let the long window absorb the imprecision. Instead of agonising over returning exactly the right three chunks, return thirty and let the model sort them out. You get retrieval's cost control and long context's tolerance for approximate recall.