Persistent LLM memory
A language model has no memory at all. It has a context window, which is a very different thing, and most of the confusion about LLM memory comes from treating the second as a weak version of the first.
The model does not forget, because it never knew
Nothing is retained between calls. Each request is answered from the weights, which are fixed, and the text you sent, which is discarded afterwards. The impression of continuity inside a chat is the interface resending the conversation each turn.
This is worth being precise about, because it determines what can be fixed. The model is not losing information; it is being sent less than you think. Anything it should know has to arrive in the request.
Bigger context windows do not solve it
Every increase in context length prompts the suggestion that memory is now unnecessary. It is not, for three reasons.
Cost and latency scale with what you send. Paying to resend a year of history on every turn to use four facts from it is expensive in a way that gets worse as the year gets longer.
Attention is not uniform. Models are measurably worse at using material in the middle of a very long context than at either end, so burying the relevant fact in a million tokens is not the same as telling the model the fact.
And a window is per conversation. Whatever the size, a new conversation starts empty, and so does a different application. The problem is not capacity, it is that nothing is stored anywhere.
What persistence actually requires
Storing text somewhere is the easy tenth of it. The rest is what makes the store worth querying.
Deciding what is worth keeping, so recall is not searching a transcript. Resolving that two mentions are one thing, so a search finds all of it rather than a third of it. Noticing that a new fact contradicts a stored one, and keeping both rather than letting the newer silently win. Carrying the permissions that came with the source, so summarising something private does not quietly make it public.
None of that is retrieval, and all of it is why a vector database on its own does not get you a memory.
Where it should live
Outside the application. A memory that belongs to one assistant is a memory you lose when you change assistants, and most people now use several.
The practical test is whether a fact captured on your phone is available in the tool you happen to be working in an hour later. If it is not, the memory is scoped to a product rather than to you.
Common questions
- Why does ChatGPT seem to remember things?
- It keeps a short set of notes it decides are worth saving, scoped to that product. It is real and it is useful, and it is not a store you can query, export, or read from anywhere else.
- Does fine-tuning give a model memory?
- No. Fine-tuning changes behaviour and style, and it is a poor and expensive way to store facts. Facts change; weights retrained on Tuesday still say what was true on Monday.
- Is a longer context window cheaper than a memory layer?
- Only while the history is short. Resending everything costs more each turn as the history grows, where retrieval returns a similar amount regardless.
- Can I move my memories between models?
- If they are stored outside the model, yes. That is most of the argument for keeping them somewhere you control: the model you use in a year is probably not the one you use today.