Your Agent's Context Window Is RAM. You're Filling It Like a Hard Drive.
The first version of my Rapidfy agent forgot the instructions I gave it on line one. Not after a week, inside a single long session. The window filled up, and the rules I set at the start slid out of reach while the agent kept working like they were never there. That is the moment I stopped treating the context window as a place to keep things. I rebuilt the agent to forget on purpose. Every iteration starts with a fresh window now, and everything worth keeping lives on disk, in git and a progress file, not in the conversation.
The context window is RAM. RAM is not storage. Most production agent failures I read about trace back to teams treating the two as the same thing.
RAM holds what the processor needs right now. It is fast, expensive, and finite, and when it fills, the processor evicts. Storage holds what you might need later, slower and cheaper and persistent. An operating system works because those two layers have clear roles and enforced boundaries. An agent that appends everything to the context window has no storage layer, no eviction policy, and no principle separating what it needs this turn from what it needed three turns ago. It fills up, gets slow, gets expensive, and starts forgetting things that were never actually removed.
The version of this that fails looks ordinary. A support agent picks up a case that runs across three days, and every tool call, every intermediate response, every user message gets appended to the window. The session opens at 2,000 tokens. By day two it sits at 25,000. Inference cost has grown linearly with the length of the conversation, which was never in anyone’s budget. The customer’s original problem, stated in the first session, is now buried in the middle of a 25,000-token context, and the agent acts like it forgot.
It didn’t forget. The fact is still in there. The architecture put it somewhere the model can’t see well.
Researchers call this lost in the middle. A model retrieves reliably from the start and the end of a long context and pays much less attention to the middle. The information is present, but long contexts have a retrieval gradient, and it drops off fast away from the edges. A complaint mentioned in turn three of a thirty-turn session sits in territory the model effectively ignores by turn fifteen. I wrote about the token side of this in context window economics for AI agents, where overhead crowds out room for reasoning. This is the other half. Even the tokens you keep are not equally visible.
Mem0’s LoCoMo benchmark puts numbers on the trade, and the numbers are not the ones I expected. The full-context baseline, where you put everything in the window and let the model sort it out, scores highest on accuracy at roughly 73%. The tiered approach, working memory plus targeted retrieval from a store outside the window, comes in around 67%. Stuffing the window wins on accuracy. It wins by six points.
Then you look at what those six points cost. Full context answers in 17.12 seconds at the 95th percentile and burns 25,000 or more tokens doing it. The tiered design answers in 1.44 seconds and spends under 7,000. Roughly twelve times faster, roughly a tenth of the bill, for six points of recall. And that ratio gets worse for the baseline on every turn you add, because its cost grows with the length of the conversation while the tiered one does not. The context window is still present in the tiered design. It just stopped being a filing cabinet for the whole conversation.
The architecture behind that result has three tiers. Working memory is the context window, fast and expensive and wiped at session end, holding the current message, the immediate task state, and the last few tool calls. Recall memory is a searchable store outside the window, usually a vector database with recency weighting, holding summaries of past sessions and facts pulled from earlier interactions. Archival memory is long-term storage for documents and logs the agent rarely touches but cannot afford to lose. When the agent needs something old, it queries recall memory and gets a targeted answer back instead of the entire history.
This is the same move I make in my Rapidfy harness, at a smaller scale. The progress file on disk is recall memory. Git is archival memory. The context window holds one feature at a time and nothing else, because an agent carrying ten features of history in its window builds the eleventh one worse.
The hard question is eviction. What leaves working memory, and when. The mistake is answering it reactively, once the window fills, by which point the cost and the accuracy are already gone. The policy has to exist before the first session runs. Tool results older than N exchanges get summarized and moved to recall. User-stated facts get written to the persistent store the moment they’re stated. Intermediate reasoning is ephemeral and gets dropped when the turn ends. Those choices are what the agent considers worth remembering, and if you don’t make them, the window makes them for you by running out of room and evicting whatever showed up first.
The second decision is to write memory asynchronously by default. A write that blocks the response adds latency the user feels directly, and a session that saves to the persistent store before returning each answer feels slow in a way that’s hard to trace from monitoring. Make the writes async and the agent answers while the save runs behind it. That is the difference between a system that feels like it’s thinking and one that feels like it’s waiting.
The cost of getting this wrong does not show up in a benchmark. It shows up as a customer explaining their situation for the third time, an agent losing the thread at a handoff, and a multi-step workflow that falls apart in the middle because a fact stated in session one was invisible by session three. The window evicted it, or buried it past the gradient. Either way the person on the other end concludes the thing is not paying attention, and they are right.
Memory is now a first-class architectural concern. There is a benchmark for it, a research literature, and a growing set of memory systems built for agent workloads. Every agent that runs longer than a single session is a memory system with inference bolted on top, and the teams building the good ones design the memory layer before they write the first prompt. The models keep getting faster, but the architecture is still the part you own. The next agent you build will remember one thing and lose another on every single turn, whether you decided which or not. The only real choice is whether you wrote the eviction policy, or the window wrote it for you when it ran out of room.
Be the first to comment
Thank you! Your comment will appear shortly, usually within a couple of minutes.
Something went wrong. Try again or contact me directly.