AI agent memory
Agent memory is one phrase covering four different things, and most arguments about it are two people meaning different ones. Sorting them out first makes the design question straightforward.
The four things people mean
Context is what fits in the prompt this turn. It is not memory in any useful sense: it is a buffer that is discarded, and managing it is a packing problem.
Working memory is what the agent keeps across the steps of a single task. Scratchpads, intermediate results, the plan it is following. It ends when the task does, and it should.
Episodic memory is what happened. This conversation, that run, the decision made on the 3rd. It is the one people mean when they say an agent should remember, and the one most systems do not have.
Semantic memory is what is true. Not that somebody said the price was 2,340, but that the price is 2,340, held separately from any one telling of it. This is where contradictions live, because two episodes can disagree about one fact.
Why a transcript is not memory
The obvious approach is to keep every conversation and search it. It works until it does not, for two reasons that arrive at different times.
The first is cost. A transcript store grows without bound and retrieval over it returns long passages, so the prompt grows with the history and so does the bill.
The second is worse and arrives later. A transcript records that something was said, not that it is true. Search it for a price and you get every mention, including the wrong figure from before it was corrected, with nothing to prefer between them. Extracting facts and keeping them separately from their tellings is what makes the second problem tractable.
The part everybody underestimates
Writing memories is easy. Deciding they refer to the same thing is not.
The customer is Northwind in the CRM, Northwind Traders in the contract, and northwind-ltd in a channel name. Three memories about one company are three unrelated rows unless something resolves them, and search will happily return one and not the others.
The same applies to contradictions. When a new fact disagrees with a stored one, an agent that overwrites is confidently wrong at exactly the moment it should be uncertain, and it has thrown away the evidence that would explain why it changed its mind. Keeping both, with their sources and dates, is more work and it is the difference between a memory and a cache.
How to add it without making a second problem
Two calls around your existing loop: query before reasoning, write the conclusion after. That is genuinely all the integration is.
Write conclusions, not transcripts. An agent that stores every intermediate thought fills its own recall with noise, and next month it is reasoning about last month's reasoning rather than about what was true.
Separate tenants from the first day. Memories written without a scope are not automatically placed in one later, so a multi-tenant agent that skipped it has a mixing problem to clean up rather than a setting to change.
Common questions
- Is agent memory the same as RAG?
- No, though they share machinery. RAG retrieves passages from a corpus you already have. Agent memory is built from what the agent encounters, has to decide what is worth keeping, and has to handle the same fact arriving twice and two facts disagreeing. Retrieval is one part of it.
- How much memory should an agent keep?
- Fewer things than you expect. Recall that returns five relevant memories beats recall that returns fifty, and a store full of an agent's own intermediate reasoning is worse than a small store of its conclusions.
- Do I need a vector database?
- You need vector search, which pgvector provides inside Postgres. A separate vector database is worth adding when your scale demands it, and it will not solve extraction, entity resolution or contradictions, which is where the work actually is.
- Can two agents share one memory?
- Yes, and it is usually the point. A research agent and a drafting agent reading one store means what one learns on Monday is available to the other on Tuesday with no handoff format between them.