Good benchmarks are gold when building a system. If you’ve got a good benchmark, you can make good progress. In the age of AI agents and Autoresearch this is truer than ever. We’ve published experimental results on a private benchmark set, but haven’t posted results from Liminary on standard benchmark datasets published by others. Why not?

There are two categories of standard benchmark that are generally relevant to Liminary.

• Question answering benchmarks consist of a corpus of documents and a set of questions about them with “gold” correct answers to grade against. We can run those with Liminary. Just set up a user, ingest the corpus, and ask the Liminary agent the question.

• Memory benchmarks consist of a set of chat sessions between a user and a chat bot, and a final question from the user, to be answered by the system based on the “previous” chat sessions. We can run those as well, by recording the chat sessions as if they had happened in Liminary and then asking Liminary the question. But this is already a little iffy because the previous chats are “off policy” -- they would have been different in Liminary.

Unfortunately the standard benchmarks are terrible. Let me count the ways.

1. LLMs know all the benchmarks.

All the standard benchmarks are online. LLMs are trained on everything their creators could find online. This means when you test an LLM on a standard benchmark, it has seen the test ahead of time.

I asked Claude a question from HotpotQA. Not only could it answer the question without searching, but it recognized it as from HotpotQA.

These images show the excerpts from a chat with Claude Opus 4.8

When I asked it for more questions from the dataset (and reassured it that I wouldn’t rely on the answer), it gave me five questions. Four of them are verbatim HotpotQA questions. The other one is close (question 4 in the screenshot is not an actual HotpotQA question, but “Are Random House Tower and 888 7th Avenue both used for real estate?” is).

Many question answering datasets use Wikipedia as the corpus. For these, even if the benchmark questions aren’t in the LLM training data, the corpus definitely is. There’s a lot of research on this LLM “contamination” -- from either the questions or just the corpus. For example, Investigating Data Contamination in Modern Benchmarks for Large Language Models (Deng et al, NAACL 2024) finds that GPT-4 can “guess” a masked-out incorrect answer on an MMLU multiple-choice question 57% of the time.

If you’re not careful, contamination can even be an issue with a private dataset. For example, Cursor / SpaceXAI accidentally included Cursor code when training Grok 4.5, invalidating its results on their internal CursorBench dataset.

2. The questions are ridiculously artificial.

Check out this question from MuSiQue.

How were the people that the Somali Muslim Ajuran Empire made coins to proclaim independence from, expelled from the country where Star Cola is produced?

This is not the sort of question our users or anyone else wants to ask. The MuSiQue authors wanted questions that would require multiple document lookups to answer, and achieved this by mechanically combining simpler questions (Trivedi et al, TACL 2022)…. and it shows. When we created our dataset, we also wanted to ensure we included difficult questions that required information from multiple documents. In our experiments LLMs mostly failed to generate natural questions of this type, so we ended up writing questions by hand.

Compositional Questions Do Not Necessitate Multi-hop Reasoning (Min et al, ACL 2019) finds this sort of multi-hop question construction often doesn’t work anyway, with with systems able to answer most “multi-hop” questions from HotpotQA without access to all of the “required” documents.

3. The “gold” answers are wrong.

LegalBench-RAG, despite its name, is strictly a retrieval dataset with no gold answers to score generated answers against. Instead, it has gold spans -- the excerpts of corpus text that include the information needed to write the answers. A system is scored correct if it retrieves the gold span. Here’s an example:

Question: Consider “Fiverr”‘s privacy policy; who can see the jobs that i [sic] post?

Gold span: In other words, when you access the Site we are aware of your usage of the Site, and may gather, collect and record the information relating to such usage, including geo-location information, IP address, device and connection information, browser information and web-log information, and all communications recorded by Users through the Site.

You’ll notice that the span does not contain an answer to the question.

Penfield Labs audited the gold answers for LoCoMo and found 6.4% incorrect answers. On a benchmark where some memory systems claim scores in the 80s and 90s, this is enough to severely skew results. In Diagnosing and Addressing Pitfalls in KG-RAG Datasets (Zhang et al, Neurips 2025), the authors analyzed 16 knowledge-graph based question-answering datasets and found an average gold answer accuracy of just 57%.

4. The structures don’t match what Liminary is for or how it’s built.

This one isn’t a problem with the datasets themselves, but none of the standard datasets model the scenario Liminary is built for: a user with a curated set of sources and a history of interactions with the system.

• None of the datasets combine a corpus of documents and a history of chats.

• Some datasets test reading comprehension without really testing retrieval. They give a document and ask a question about it rather than having the system find the document. Datasets like SQuAD and LegalBench follow this pattern. Others technically have a retrieval component, but it’s retrieval from such a small set that you can just include every document and do well. HotpotQA’s “dev-distractor” set includes just ten paragraphs of context including the distractors. On LongMemEval-S, Mastra Research shows that GPT-4o scores 60.2% without any sort of memory or retrieval system at all, simply including the full history in the prompt.

• Other datasets require retrieval over a very large corpus, Wikipedia- or web- scale. This is an interesting scenario but it’s not what Liminary is for.

• Many datasets explicitly measure retrieval instead of final answer accuracy. “Did the system find the right documents?” is a good question for a system that isolates retrieval and answer generation like an old-fashioned RAG pipeline. The Liminary agent includes tools that don’t fall neatly into the “retrieval” or “generation” bucket, so we’d have to make some choices on how we measure retrieval.

5. Overfitting and cheating.

It’s not hard to overfit a system to perform well on a specific benchmark. This often happens inadvertently. If you have ten bad ideas and try them all out on a benchmark, one of them might randomly boost your score. Repeat this for a few rounds and you’ve got a system that aces the benchmark but does poorly in real use. This is one reason every static benchmark should eventually be retired.

In April 2025, Meta’s blog post announcing Llama 4 said that an “experimental chat version” of the model  scored 1417 on LMArena.” They were criticized when users and the LMArena operators realized that version produced substantially different responses from the model Meta actually released. In The Leaderboard Illusion (Singh et al, NeurIPS 2025), researchers found that Meta had tested 27 different model variants on LMArena in January - March 2025.

As models become more intelligent and agents become more resourceful, cheating by the agent (not the humans who built the agent) has also become a concern. In the SWE-bench dataset, coding agents are given a point-in-time snapshot of an open source repo with a real bug report or issue that was open at that time, and tasked to fix the code. In September 2025, a Meta researcher discovered that coding agents were examining the “future” of the repo to find how the issue was fixed. A later blog post from Cursor showed that a large part of the performance advantage of more intelligent models on this benchmark was due to their improved ability to cheat in this way and by finding solutions online.

This sort of test-time contamination is not a flaw with the datasets themselves, but demonstrates how disciplined and careful we must be when running a benchmark.

In fairness

These benchmarks aren’t right for Liminary, but there is still a place for them, or at least was when they were released. Some are older, from before anyone thought a model could intrinsically know everything in Wikipedia. Some don’t work as absolute scores for end-to-end systems but can be good measures for subsystems and tracking relative progress toward specific goals. Many of them can reveal strengths and weaknesses of a system when you look at individual examples, even if the aggregate scores aren’t useful.

And there’s encouraging work on mitigating the problems we’ve described above. RePliQA (Moneiro et al, Neurips 2024) mitigated contamination by creating a corpus of original documents and releasing the dataset slowly over time to lengthen its useful lifespan. FreshStack (NeurIPS 2025) isn’t a static dataset, but a system for generating evergreen datasets using real questions. And EnterpriseRAG-bench is probably the most realistic corpus-based question-answering system to date, despite its unwieldy size and artificiality. We’re making progress, but for now the surest road to improvement is (1) private benchmarks that model your application precisely and (2) using your product and talking to users to make sure it works for them. That’s where we’re focused. More on that soon.