In January, a customer gives a retailer’s agent her address: 14 Alder Row. In July she moves across town to 7 Wrenfield Road. She never updates her profile; she just mentions it to the agent in passing.
In August she places an order. The memory layer returns Alder Row: top match, genuinely hers.
The parcel goes to her old flat.
No check fires. Same city, valid address, matching name. Nothing in the system knows the record is stale.
Retrieval did its job. The agent found a true fact about the past and treated it as a fact about the present.
Agent memory has outgrown retrieval.
What an agent remembers differs not only in relevance, but in whether it is still true and what authority it carries. Similarity search is good at the first. It does not, by itself, resolve the other two.
Retrieval reconstructs the past. An agent often needs to construct the present.
Finding a memory isn’t knowing what’s true
A common memory pattern is still inherited from RAG: persist outside the context window, embed, retrieve top-k, inject.
Store → Embed → Retrieve → Prompt
For many workloads that is the right design. Now look at what the agent actually held:
January address = 14 Alder Row
July address = 7 Wrenfield Road
Both records are relevant, and both are genuinely hers. Nothing in retrieval says which one is current. An agent about to ship a parcel needs something stronger than recall:
14 Alder Row superseded
7 Wrenfield Road current
The task isn’t retrieve the customer’s address. It’s resolve her current state.
Once agents act on systems that change, this is everywhere. A preference changes. An order moves from processing to shipped. A manager revokes an approval. A policy is replaced. These aren’t retrieval events. They are state transitions, and a memory layer has to represent them as such.
This sounds academic until the agent can do something.
Then it’s operational.
The problem kept moving
This isn’t a production quirk. The literature has been moving in the same direction, one problem at a time.
First, we needed somewhere to put information that no longer fit in context. MemGPT’s lasting contribution was the separation it made explicit: what an agent can remember and what the model sees right now are different things. Lost in the Middle showed why that separation matters: a longer window doesn’t guarantee the model uses what’s in it.
Then we realised that storing everything wasn’t useful, so memory formation became selective. Mem0 moved the effort to write time: decide what is worth keeping, and update rather than append.
Then we found that even well-curated memories lose something when treated as independent chunks: their relationships.
Take four events in the same customer’s history. She reports a failed refund. A manager approves a retry. The payment provider rejects it. She contacts support again. Similarity will rightly surface all four. It won’t encode that they are ordered, share an entity, and may be causally linked. The rejection changes what the approval means.
Similarity finds the pieces. It doesn’t find the story.
A-MEM moves memory toward an evolving network of linked notes. MAGMA takes this further, keeping semantic, temporal, entity and causal views separate and letting the query choose the traversal. A when question and a why question shouldn’t take the same path.
The lesson I take from this isn’t use a graph. It’s that a memory architecture shouldn’t force every question through the same retrieval primitive. A graph won’t make an inference true. It can make a wrong inference more durable.
Current still isn’t the same as authoritative
There are already useful approaches to the address problem. Temporal memory systems such as Zep’s Graphiti attach validity intervals to facts, so the January address is invalidated when the July one arrives.
That handles what is current. It doesn’t handle what comes next.
The same customer asks for a pink statement necklace for a friend’s wedding. A good answer draws on four kinds of information:
- Semantic context. Magenta belongs in the pink family; “statement” is a style. Some of these links were inferred by a model.
- User memory. She once said she prefers silver.
- Live data. The necklace she’d pick has five units left.
- Policy. A business rule limits which products and promotions the agent may offer.
All four land in the same prompt. None of the risks here is about which record is newest.
A remembered inventory count could outrank the inventory system because it scored well. A colour link the model inferred could carry the same weight as the merchant’s catalogue. Her preference could quietly start behaving like enterprise truth. And the policy could drop out of the prompt because another memory looked more relevant.
These are problems of authority: which source is allowed to win. And of lifecycle: how each kind of information is allowed to change.
She can revise her preference in a sentence. Only the business can change a policy. A live stock count shouldn’t be treated as durable memory or as the source of truth.
The sharpest version of this problem I’ve seen is in order flows. Once a cancellation request is accepted, the order system moves the item into a cancellation-in-progress state: it is no longer cancellable, and the quantity that can be cancelled drops to zero.
Seconds earlier, the agent had correctly observed that the item could be cancelled. If it keeps that as memory and retrieves it later, retrieval has worked. The agent now offers an action the system no longer permits.
So in the system we’re building, the boundary is simple. Memory can record that the item was cancellable and that a cancellation was requested. Whether it can be cancelled now comes from the order system, every time.
The agent needs one working context. The systems that feed it don’t need to be one system.
What I would build
Before deciding where memory lives, I’d ask five questions.
What am I remembering? An event, a preference, a model inference, a completed action, a policy, a failure not to repeat? These are different objects.
Can it change? If so, history and current state need to be separate. Overwriting loses the history; appending leaves the model to guess which record is current.
Where did it come from? User input, system-of-record data, tool output and model inference don’t deserve equal trust. Provenance belongs in the schema from day one.
Which relationships matter? Sometimes similarity is enough. Sometimes the useful relation is temporal, entity-based, causal or task-dependent.
Does the model need it now? Persistent memory and working context are separate decisions.
From those answers, a few commitments. I’d keep historical evidence separate from current-state projections. I’d never let hard constraints compete with optional memories in the same top-k. And I’d build expensive structure asynchronously, rather than putting a model call in every synchronous write.
Now the limits. Over a mostly static document collection, vector retrieval may be exactly right. For a handful of stable preferences, extraction plus retrieval is probably enough. You don’t need a temporal knowledge graph just because one exists.
The machinery earns its place when truth changes, actions have consequences, tasks persist, or information carries different authority.
The architecture should follow the failure mode.
The platforms are drawing the same boundary
Agent platforms are increasingly decomposing the same problem into separate concepts: sessions, memory, artifacts, checkpoints, compaction and context.
Google’s ADK frames context as a compiled view over a richer stateful system. Anthropic frames the work as context engineering. OpenAI has moved compaction into the agent runtime; Microsoft treats workflow checkpoints as explicit durable state.
The interesting part isn’t the terminology. It’s the boundary being drawn between persistent state and what the model sees now.
Retrieval didn’t disappear. It changed jobs.
None of this means RAG is dead. Semantic retrieval remains one of the most useful tools we have. What changed is its role: it used to be treated as if it were memory. Now it is one step in a longer lifecycle.
The working definition I’ve landed on:
Agent memory is the machinery that maintains useful state across time and controls how that state is allowed to influence what the agent does next.
The harder question comes after remembering
Once information persists, the question becomes what the system is allowed to do to it.
Can it be summarised? Can its influence fade? Can it be retired without being deleted? What happens when two memories contradict each other? And what must never be compressed far enough to change its meaning?
A newer line of memory research is looking at failures that happen before retrieval is ever invoked. The Compaction Cliff shows what happens when repeated compaction treats a safety constraint like ordinary history. WhenLoss asks whether important information is already lost during memory formation, before retrieval ever gets a chance to fail. Weighted Memory Tree separates what stays stored separates what remains stored from what is allowed to keep influencing the agent.
Adding another reranker won’t recover information you destroyed when you saved it.
So the question for Part 2 isn’t can the agent remember this? It’s what should the memory system be allowed to forget, change, compress, or stop listening to?
