An update on how we measure our memory system: what moved, what didn't, what we learned about the benchmarks themselves, and why we publish our readings while the work is still in progress.

We wrote about why we're skeptical of AI benchmarks as a way to compare products. Contaminated training data, artificial questions, wrong gold answers: we still believe all of that.

But there's a second use of benchmarks that we lean on heavily: steering our own work. When you change a retrieval pipeline or rewrite a prompt, you need something that tells you whether you actually made things better or just different. For that job, a benchmark you run yourself, on fixed questions, with a fixed judge, is genuinely useful, not as a prize to be won but as another system metric to monitor and improve. In this post we cover three memory benchmarks we evaluated, what we learned from those evaluations and how the process more than the final number turned out to be the most valuable piece of this exercise.

LongMemEval: real gains, and an honest look at the error bars

LongMemEval tests whether an assistant can answer questions about things a user said across hundreds of past chat sessions. Around 115K tokens of history per question, 500 questions, six types: what the user said in a single session, what the assistant itself said, the user's stated preferences, aggregation across sessions, temporal reasoning, and knowledge updates. A slice of the questions is also modified so the history genuinely doesn't contain an answer, to check that the system refuses instead of guessing.

Our first clean baseline, plain retrieval over past chat history with no memory system at all, scored in the high eighties on the 100-question sample we used early on for quick iteration. What this says about the benchmark is that a solid retrieval pipeline gets you most of the way, and the remaining points are where a memory system adds value. Ours brought us into the low nineties over a couple of months.

This cycle we fixed a few bugs in the answer pipeline and re-ran the full 500 questions. The best run came back at 93.4%.

As far as we can tell it sits above the best result we've seen with its methodology disclosed (91.4%, from a vendor's technical report) and above the range the benchmark's own authors measured when they handed models the evidence directly, with no retrieval at all (roughly 87 to 92%). Some vendors report higher numbers than ours, but without the methodology needed to interpret them: no run counts, no variance. If this post shows anything, it's that those details can move a score by more than the gaps between vendors.

We haven’t seen a lot of re-run related data published so we wanted to share this in the hope that this helps with interpretation. After that run, we ran it again with the system code untouched: same questions, same judge. It scored 91.6%.

Twenty-three of the 500 questions flipped between two runs of the same system. That's a spread of almost two points between the two runs, from nothing but the ordinary nondeterminism of a live pipeline. We checked how results in this space are usually reported, and as far as we can find, nobody publishes run counts or variance. Every number you've seen on this benchmark, including our old one, is a single roll of the dice. So: our best run is 93.4%, our band across repeat runs is 91.6 to 93.4, and the average being 92.5.

Temporal reasoning went from 90.2% to 94.0% in both runs exactly. Knowledge updates went from 94.9% to 98.7% and then 100%. Those are the two categories our fixes were aimed at, and they moved the same way twice. The biggest discrepancy in the run-to-run results is in the 30 preference questions, where the score moved 16.7 points between back-to-back runs; multi-session wobbled too, by less.

Why preference questions are so noisy

Preference questions are graded against an open-ended rubric ("a good answer builds on what the user told you about themselves"), and an LLM judges compliance. We score every run with our default judge, a model from the GPT family, deliberately a different family from the default model that writes our answers. As a check, we re-graded the same 500 answers with a second judge from the Anthropic family: 92.2%, close to the first judge overall. Category by category the two mostly track each other except for preference questions, where they differ dramatically.

Here's one example, so you can see what the judges see. The user once mentioned buying a portable power bank. Later they ask:

Our system came back with a full battery-care guide: five sections of bullet-point advice covering display settings, background activity, connectivity, system features, and charging habits. The very last bullet reads:

That's the one personalized touch. Is one bullet at the end of a generic guide a good personalized answer?

Both of these assessments are defensible, but neither of them is a fact. When a benchmark's score partly depends on which reasonable reading a judge takes, decimal places stop meaning much, and that's worth knowing whether the number is ours or anyone else's.

We've outgrown this benchmark

This is the uncomfortable conclusion of the exercise: LongMemEval can no longer tell us whether we're improving. Our band sits at or above the range the authors measured with retrieval taken out of the picture entirely. Of the failures that remain, a meaningful share are questions where two judges disagree, or where we believe the gold answer itself is wrong (one has a name in the answer key that appears nowhere in the conversation history). Our room for improvement is smaller than the noise level. It just means we need harder tests, and that’s why we looked at other benchmarks.

Memora: measuring forgetting

Remembering is half of memory. The other half is letting go: facts get corrected, preferences change, and users explicitly tell an assistant to forget things. A memory that keeps surfacing retracted facts is worse than no memory at all. Memora is the first benchmark we've found that makes this the headline metric: its FAMA (Forgetting-Aware Memory Accuracy) score rewards recalling what should be recalled and penalizes reliance on what should be forgotten.

We ran it on our production configuration: the same retrieval settings, the same models, the same agents our users hit every day. We kept it unchanged because we want the score to describe the product people actually use, not a benchmark-special build. The one accommodation: the benchmark's forget instructions are applied by the harness as a scripted step rather than through weeks of live conversation, which is worth disclosing since forgetting is the headline metric. That also means the numbers below are our own baseline, not an entry in the paper's results table; we're not claiming a ranking.

Underneath the composite sit two questions: did we surface the facts we should have, and did we avoid leaning on the facts we were told to drop? The second one, forgetting accuracy, is 75.0%.

That number is the reason we ran this benchmark, and it produced the most useful finding of the batch, precisely because nothing moved: the same pipeline fixes that helped on LongMemEval lifted Memora's overall score too, but forgetting didn't budge. On recommendation-style questions, close to two in five forgetting checks still fail; the system retrieves nearly everything relevant and doesn't yet reliably suppress the parts it was told to let go of. We wouldn't have that sentence without the benchmark, and it's now at the top of our memory roadmap.

Caveats: this is a single run of 150 questions, so the error bars are wider than our run of LongMemEval at 500.

BEAM: no number yet, but some findings worth sharing

The largest history LongMemEval ever tested us on is about 115K tokens, a few hundred chat sessions. Liminary is built to remember everything you've said and saved across months of use, which is far more than that. So we've started running BEAM, a new benchmark of coherent conversations from 128K up to 10 million tokens, with ten distinct memory abilities per conversation.

We have internal numbers on the smallest tier. We kept BEAM's judging prompt byte-for-byte but put our default judge model behind it instead of the older model the benchmark pins. A newer judge simply grades more reliably, and since we run benchmarks to steer our own system rather than to fill in a leaderboard, we chose better judgments over comparable ones. The cost of that choice is real: BEAM's published numbers come from its pinned judge, so a headline from ours would invite a comparison it can't support, and the rest of our harness is still being vetted. What we can share now is what we learned building it, which we think is more useful anyway.

Our acceptance test ran backwards. The paper's own best system scores 0.358 on the tier we ran, and two of the ten categories are nearly unwinnable as scored (no published system cracks 0.21 on either of them, and on one nothing beats 0.05). So when we validated our harness, a high score was the red flag: anything above roughly 0.5 most likely meant an aggregation bug in our scoring code, not a great memory system.

The published scorer has two quirks that change results. Reading the reference implementation, we found that a placeholder for the question text is never actually filled in when grading (the judge is asked whether an answer addresses a question it is never shown), and that partial-credit scores of 0.5 are silently rounded down to zero in nine of ten categories. We reproduced both faithfully in our harness, since comparability is the point, and we also measured what fixing them changes: repairing the placeholder alone shifts nine of the ten category scores, the largest by 16 points. Anyone comparing numbers on this benchmark is comparing through those quirks.

Most third-party numbers can't be compared at all. Some vendors report scores around 0.73 to 0.79 on this benchmark, roughly double the paper's best published system, without disclosing their judge, prompts, aggregation, or question counts. One of them, Cognee, deserves real credit here: their own writeup discloses that their headline came from a single conversation out of twenty, with per-question-type prompts, and says the number "should be taken with a grain of salt." That's the disclosure norm we'd like to see everywhere, ours included.

Where this leaves us

Our memory system got measurably better this cycle in the places we can prove it: temporal reasoning and knowledge updates on LongMemEval, replicated across runs. Memora's overall score moved the right way too. And we know exactly where the next gains are: suppressing retracted facts in recommendations, now at the top of the roadmap. We've retired one benchmark because we've outgrown it, brought a second into rotation precisely because it points at that headroom, and are still calibrating a third. We'll keep publishing the readings, error bars and our learnings along the way.

Earlier in this series: On AI benchmarking · It's the harness, not the model