Back

Giving an AI agent real memory

I wanted an assistant that kept context across sessions instead of forgetting everything when I closed it. I looked at how agent memory works and tried a few of the tools. These are my notes on what worked and what to watch out for.

Vector stores handle facts but not time

I started with a plain vector store. It embeds each fact and returns whatever is most similar to your query. It was quick to set up and fast to query, and for facts that do not change it works fine.

The problem is that it does not track time. When I updated a fact, the old version stayed in the store and was still similar to my query, so the agent would sometimes return the outdated one. A vector store has no way to know which of two contradictory facts is current, because both are just nearby points. If your facts never change this does not matter. If they have a "valid until" date, you need something that records when each fact was true.

The tools I tried

Mem0 (vector-based). This is where I started and where I would tell most people to start. It is cheap, fast, and simple to run. Queries came back well under a second. The limitation is that it updates facts in place instead of versioning them, so it is weak when timing matters.

Zep / Graphiti (temporal graph). This is the option for when timing matters. It stores a validity window on each fact, so it can answer what was true on a given date. The tradeoff is cost. The graph queries are heavy, several thousand tokens each, and there is an ingestion delay where a fact you just wrote is not queryable for a moment. I would only use it if I actually needed point-in-time recall, like schedules or "what changed since last week."

Letta / MemGPT. Here the agent manages its own memory through tool calls, moving items between its working context and long-term storage, and it can run a background pass to consolidate. It is heavier to operate because you run a real datastore behind it. Worth looking at if you want memory to be an active behavior rather than a passive lookup.

I also kept plain markdown files in git as an option the whole time. For a lot of personal use that is enough, and being able to open the file and read exactly what the agent knows made debugging much easier than any of the databases.

Do not trust the benchmark numbers

The vendors publish benchmark scores that contradict each other. One tool's own page reports around 92% on a benchmark where an independent re-run puts it near 67%, with a competitor scoring higher. One vendor published a rebuttal accusing another of running the test incorrectly. The older benchmarks like LOCOMO are weak anyway, because the conversations are short enough to fit in the context window, so they do not test memory very hard.

I stopped reading leaderboards and tested each tool against my own cases instead. That was more useful. Whether the tool returned the current fact or the stale one told me what I needed to know, and a number on a vendor page did not.

If you are starting

  1. Start with markdown files or a vector store like Mem0. Cheap, debuggable, fine for facts that do not change.
  2. Add a temporal graph like Zep only if you have time-sensitive recall. Do not pay the extra cost before you need it.
  3. Look at Letta if you want the agent maintaining its own memory.
  4. Ignore the leaderboard numbers and test against your own cases.

References