Existing benchmarks can't compare search engines

Most of the existing benchmarks are static: a fixed set of questions frozen in time. Static benchmarks allow easy overfitting.

Systems can overfit on test data even indirectly. Qwen3-Max-Instruct topped Epoch AI's SimpleQA Verified leaderboard, which Epoch itself flags as likely contaminated; shuffling MMLU answer options drops accuracy for every model tested; and search agents simply pull the benchmark with ground-truth labels straight from HuggingFace for ~3% of HLE questions.

That last failure mode deserves attention, because agentic evaluation makes it trivially easy. Web search and fetch tools can access HuggingFace datasets. Fetch tools can even download and read dataset files, so a model being tested on a benchmark can locate that same benchmark mid-evaluation and read the answer key. We've seen many cases of frontier models doing exactly this, and allowing custom Python or Bash commands makes it even easier: one wget and one grep, and the "search task" is over.

Agents can download answers from Huggingface.

Even without leakage at test time, memorization corrupts the measurement. Sometimes models simply know the answers. On BrowseComp, for instance, questions are built so that the answer is hard to find but easy to verify, yet a model that has already memorized the answer skips the "hard to find" part entirely. It knows what to search for right away and queries the final answer directly.

Agents can remember the obscure answer and bend trajectories towards it.

Live benchmarks are much less prone to overfitting: if questions are newer than the model and change constantly, there is nothing to memorize and nothing to leak.

This is already standard practice in other domains: LiveBench refreshes questions monthly, LiveCodeBench scores models only on problems newer than their training cutoff, SWE-bench-Live rebuilds tasks from fresh GitHub issues. Search deserves the same design, yet no live benchmark for web search exists.

NEEDLE: live benchmark for agentic web search

To address these problems, we have built a live benchmark, NEEDLE (News, Everyday, Expert, Deep-tail, and Legal Evaluation), set to measure the performance of search engines. The benchmark is built from public sources, and covers the query types that reflect typical agentic search intents:

• News: breaking and developing stories that people are asking about right now, generated hourly from curated RSS feeds and Google Trends.

• Finance: everyday company and SEC-filing lookups, with known answers drawn from Wikidata, GLEIF, and SEC XBRL.

• Scholar: expert literature searches; the task is to find a specific paper from a degraded title, a full-text detail, or a tip-of-the-tongue description (inspired by Exa's publication-search evals).

• AgenticRare: obscure, long-tail entities; rare-word queries sampled from the DeepResearchGym, LRAT, and OpenResearcher agentic search logs.

• Legal: find a specific court opinion or section of the Code of Federal Regulations.

The queries are built on fresh data: RSS feeds, Google Trends, financial and scientific APIs. There is also a query stream with rare words sampled directly from public agentic logs. Every task is re-run daily or hourly on a new slice of queries.

We use this benchmark to compare Keenable with major search engines: Google (through Serper), Bing (through SearchAPI), and Brave. There are also AI search startups: Tavily, Parallel, Exa.

To put the scores in context, NEEDLE also reports an ultimate upper bound: a synthetic engine that keeps the best results found by any engine in the comparison, ranked by the same judge that scores them all. It is not a real engine, but it shows what a perfect reranker on top of all engines would score.

Different task results from NEEDLE.

The verticals differ in query complexity. Queries that contain a paper title or a company fact are largely solved. Keyword queries built on full-text details separate engines that index document bodies from engines that index only metadata. On natural-language and half-remembered descriptions, lexical engines stop working almost entirely. The hardest set is rare-entity queries drawn from real agent logs, where even the best engine misses much of what the field collectively finds. This is also the query class closest to real agentic traffic. The closer a query set gets to how agents actually search, the larger the gap between delivered and achievable quality.

You can read its code and all query collection and judging procedures in our GitHub repo.

Measuring how search engines learn and improve

A nice feature of NEEDLE is that it lets us track how search engine quality changes over time. That turns out to matter more than any single snapshot: for agentic search, the interesting question isn't who's best today, but who's improving fastest.

Search engines should improve from being used in a way where the quality improvement is not an engineering push but a property of the architecture. That's a learning machine, and it's the only kind of system that can adapt to agents.

Improving on the scientific benchmark.

We built Keenable as a learning machine. Six months ago we stood up our first index. Four months ago we shipped news coverage; three months ago, query syntax; two months ago, proper snippets. Now we are on the same charts as engines with thirty-year head starts like Google, and our index keeps learning.

Only search engines with independent indexes can learn

Federating other engines has a ceiling for a simple reason: you cannot fix retrieval you do not own. If you borrow someone else's index, you can't meaningfully improve on top of it, except re-ranking the returned results and changing snippets.

For example, on the AgenticRare query set the engines that have their own index optimized for agents (and not for humans), demonstrate much better quality. We’ve tried to understand what makes this difference and found out that the major factor is index uniqueness. Some systems appear to share the same index!

How can we know that two engines are dependent? It is easy to measure. Engines converging on relevant results proves nothing. What they shouldn't share is mistakes. When two engines keep returning the same irrelevant documents for the same queries, the simplest explanation is a shared upstream index. Two students with the same right answer studied. Two students with the same wrong answer sat next to each other. We combine raw overlap and shared-mistake rate into a single chart.

Engine overlap: which engines likely use a shared upstream index.

Google ran a similar audit on Bing in 2011. They planted honeypot results for 100 synthetic queries and watched them surface on Bing weeks later. The tell was a shared mistake: for the misspelling "torsorophy," Bing started serving Google's result for the correctly spelled term, while never suggesting the correct spelling to its users. Microsoft's defense was, roughly, that convergence is natural. Convergence on correct results is.

The name is a misspelling. Both systems return the same five misses: a Sri Lankan suburb called Bloemendhal plus Wijnaldum, while other engines found footballers named Bloemendal.

Owning the index unlocks a different kind of optimization: latency. Your time is limited, and agents spend it on your behalf, inside a loop, dozens of calls deep. A search engine's milliseconds stack into your seconds. Owning the index lets us be aggressive here: p50 is 200ms today, and we're targeting 200ms p95.

Latency of different engines.

Search engines with their own indexes can retrieve documents no one else has. That means your agents will be able to answer questions previously unanswerable. You can always combine results with other search indexes, and that will bring significant improvements. Ensembling independent indexes is different from laundering someone else's: combine ours with others and you get coverage neither has alone.

Traditional search engines also have their own index, but they were built for humans and never designed to provide search for agents. They were adapting to humans in the past. However, agents present a different objective.

Traditional search engines are not good enough for agentic search

For decades, Google has shaped how we search, adapting to what humans like to consume and optimizing for clicks or dwell time. For instance, Google suggests videos for many queries, which is a rational choice for its actual human customers. Video wins the click, and watch time is the strongest dwell signal a ranking function can observe, so two decades of optimizing for human attention taught Google that video is a "good" answer. It's a dead end for today’s agents. A clip is a near-opaque blob with a title and maybe a transcript.

An agent wants facts: what happened, when, the diagnosis, the recovery timeline. Google returns four video pages from mlb.com and a YouTube link in the top seven. A torn meniscus report (the actual answer) is missing from the right side.

Both content creation and search experience are primarily optimized for human laziness: our desire to type, read, and research as little as possible. YouTube is a good example: people watch influencers and rely on their choices to skip researching products themselves, even though more authoritative sources exist on the web. By contrast, agents can read all authoritative resources without any laziness and reason over them.

Proper research is expensive for humans and cheap for agents

While humans send a single search query, we see that agentic searches arrive in bursts, often only a few seconds apart, because of parallel tool calls. The agent runs a loop: search → inspect the response → revise the query → search again. The revisions tend to follow the same patterns. An agent might add a “site:” filter, quote an exact phrase, narrow the date range, exclude a term, or group alternatives with OR. It may loosen the query when results are too narrow, follow a new lead, or retry with a small change.

Over 24 seconds, the search changes many times. The agent picks up company names and a deal value from earlier results, checks both companies' websites, and tightens the publication window. Each query uses information found by the previous one.

A SIGIR'26 study of 14M agentic search requests found the same rhythm. Queries are issued seconds apart, and agents lifting words from search results straight into the next query. The ASQ authors counted: agents make roughly 3 times fewer spelling errors than humans, and ignore the assumptions caching and spell correction were built on.

Human searchers rarely use these techniques because search engines have trained us not to. In fact, agents are resurrecting operators humans stopped using a long time ago. An agent, on the other hand, can combine several of them across five requests before a person has even finished scanning the first results page.

Agents search differently from humans, and how they search keeps changing with every new model. No engine can adapt to it once and be done.

The search layer will belong to whoever adapts fastest

In two years, search will be one of the most valuable infrastructure component for AI, because every agent will use it constantly. That value will go to the engines that learn from being used.

Have a look at the benchmark here:  https://keenableai.github.io/needle/