Users say they want an AI assistant that "remembers them." Product turns that into a ticket that says "add memory." Engineering reads the ticket and reaches for the context window, because that is where the model reads from. Six weeks later the system is slow, expensive, occasionally leaks one user's data into another user's session, and still forgets the thing the user actually cared about.
The problem is that "memory" is being treated as a model capability when it is a system design problem. The model has no memory. It has a context window, which is a fixed-size input it re-reads from scratch on every single call. Everything that feels like memory - continuity across sessions, recall of a preference stated last month, knowing not to repeat itself - is architecture you build around a stateless model. Getting it right means making deliberate decisions about storage, retrieval, expiration, isolation, and relevance. Getting it wrong usually means stuffing more text into the context window until the bill or the latency forces a stop.
Short-term context is not long-term memory
These are two different systems, and conflating them is the root of most memory failures.
Short-term context is the working set for the current interaction: the last several turns of the conversation, the document the user is currently editing, the immediate task state. It is small, it is hot, and it lives in the context window directly. It expires when the session ends, and that is correct behavior. You do not want yesterday's half-finished thought bleeding into today's task.
Long-term memory is durable knowledge that should persist and be recalled selectively: the user's name and role, stable preferences, facts they established weeks ago, a summary of past projects. This does not belong in the context window by default. It belongs in a store outside the model - a database, a vector index, a key-value store - and it enters the context window only when it is relevant to the current turn, retrieved on demand.
The naive design collapses these two. It keeps appending everything to the conversation history and passes the whole thing back on every call. This works beautifully in a demo, where the conversation is ten turns long. It falls apart in production, where a power user has a conversation thread that is 400 turns long, and every message now costs you the full history in input tokens.
The cost curve of naive context stuffing
Here is the arithmetic that the "just put it in the context" approach ignores. The model re-reads its entire input on every call, and you pay for every input token every time.
Suppose each conversation turn averages 500 tokens, and you keep the full history in context. On turn N, the input includes all N-1 prior turns. The cost of the conversation is not linear in the number of turns. It is quadratic.
tokens_processed_over_conversation = sum over t of (t * 500)
= 500 * N * (N + 1) / 2For a 10-turn conversation that is about 27,500 token-reads. For a 100-turn conversation it is about 2.5 million token-reads. The conversation got 10 times longer and the cost went up roughly 90 times. A single dedicated user does not cost you a bit more than a casual one; they cost you two orders of magnitude more, and they are exactly the users you most want to keep.
Latency follows the same curve, because time-to-first-token scales with input length. The experience degrades precisely for your most engaged users, which is the worst possible group to punish.
The fix is not a bigger context window. A bigger window just raises the ceiling on how expensive a single call can get. The fix is to stop treating the context window as storage.
The pieces of an actual memory system
A memory system that holds up has five parts, and each one is a decision you have to make explicitly.
1. A store outside the model. Long-term facts live in a database or vector index keyed by user. The context window is a workspace, not a filing cabinet. You write memories to the store as they are established and read them back selectively.
2. Retrieval, not inclusion. On each turn, you retrieve the handful of memories relevant to the current query rather than loading everything. This is the same retrieval discipline as RAG, applied to user history. The question is never "what does this user know," it is "what does this user know that matters right now."
3. Relevance scoring with recency and importance. Not all memories are equal. A useful scoring function blends semantic similarity to the current query, recency, and an importance weight. A common shape:
def memory_score(memory, query_embedding, now):
similarity = cosine(memory.embedding, query_embedding)
age_days = (now - memory.created_at).days
recency = 0.98 ** age_days # gentle decay
importance = memory.importance # 0..1, set on write
return 0.6 * similarity + 0.2 * recency + 0.2 * importanceYou then take the top K by score, subject to a token budget. The weights are tunable, and you should tune them against real recall failures rather than guessing once and forgetting.
4. Expiration policies. Memory that never expires becomes memory that lies. A user's job title changes. A stated preference gets reversed. If you keep every fact forever with equal standing, the model will confidently recall stale information. You need policies: time-based expiry for volatile facts, supersession where a newer fact about the same attribute replaces the old one, and a way to delete on request. Treat this as a first-class part of the design, not a cleanup job you add later.
5. Hard user isolation. This is the one that turns a performance problem into an incident. Every read and write to the memory store must be scoped to a user identity that comes from the authenticated session, never from anything the model produced or the user typed. If your retrieval query can return another user's memories, you have built a data breach with a friendly chat interface. The failure mode is subtle: a shared vector index without a mandatory user filter will happily return the semantically nearest memory, and the nearest memory to "what did I order last week" might belong to a different customer.
A checklist before you ship "memory"
Run through this before you tell anyone the assistant remembers them.
- Is long-term memory stored outside the context window, in a durable store keyed by user identity?
- Does every memory read and write carry a user filter derived from the authenticated session, enforced at the query layer, not the application layer?
- Do you retrieve a bounded set of relevant memories per turn rather than loading the full history?
- Is there a token budget on assembled context, and do you know what gets dropped when it is exceeded?
- Do memories have an importance weight and a recency decay, and have you tuned them against real recall misses?
- Is there an expiration and supersession policy so stale facts do not resurface as current truth?
- Can a user see what the system remembers about them, and delete it?
- Have you load-tested a 200-turn conversation and looked at the per-turn cost and latency curve, not just a 10-turn demo?
- Do you have a test that attempts cross-user retrieval and asserts it returns nothing?
If you cannot answer yes to the isolation and expiration items, you do not have a memory feature. You have a liability with good demo energy.
The reframe
Memory is not something the model does. It is something you architect around a model that does not remember. The context window is short-term working memory with a hard size limit and a per-token price. Long-term memory is a retrieval system you build, with the same care you would give any other data store: scoped access, lifecycle policies, relevance ranking, and a cost model you actually understand.
The teams that get this right stop asking "how do we make the model remember" and start asking "what should persist, for whom, for how long, and how does it get recalled." That second question has engineering answers. The first one has only bigger bills.
If you are designing or repairing the memory layer of an AI product and want it to survive contact with real users and real usage curves, that is work we do. See /services/ai-reliability.