Vector search dominates AI retrieval headlines, but the moment an agent has to pull a specific SKU, error code, or contract clause, a decades-old algorithm quietly outperforms embeddings. BM25 ranks documents by term frequency, term rarity, and length normalization, and those three signals happen to match exactly what enterprise agents most often query.
Major engines, including Apache Lucene, Elasticsearch, OpenSearch, and Azure AI Search, default to BM25 as the precision layer beneath hybrid retrieval. If you are building a system that needs to reach into enterprise data, BM25 is often the retrieval mechanism most likely to find the exact token your agent needs.
TL;DR BM25 ranks documents using term frequency, inverse document frequency, and document-length normalization. It excels at exact-match retrieval for SKUs, error codes, clause numbers, and API endpoints. BM25 is deterministic, explainable, CPU-friendly, and the lexical layer in most hybrid search systems. It cannot understand synonyms or meaning, so semantic retrieval fits paraphrased queries better. Want to see lexical retrieval in action on your own enterprise data? Try Airbyte Agents and connect your first source in minutes.
What Is BM25 and How Does It Rank Search Results? Best Match 25 (BM25), also referred to as Okapi BM25, is a ranking function that scores how relevant each document is to a query by combining three signals:
How often the query terms appear in a document How rare those terms are across the whole collection How long the document is in relation to the average Documents that contain query terms more often score higher. Terms that are rare across the collection count for more than those that are common. Long documents get scaled down, so word count alone does not inflate their relevance.
The algorithm comes from the probabilistic retrieval framework developed in the 1970s and 1980s by Stephen E. Robertson, Karen Spärck Jones, and others. Lucene documents BM25Similarity, and Lucene-based engines expose BM25 in their scoring settings. The "Okapi" prefix comes from the Okapi information retrieval system built at London's City University in the 1980s and 1990s.
One point matters before we go further. BM25 is a bag-of-words function. It looks at which query terms appear and how often, but it ignores word order, phrase structure, and proximity. A query for "cosmos" will not match a document that only says "space," because the two words are different tokens. That limitation is the source of both its strengths and its blind spots.
What Is the BM25 Formula? The BM25 score for a document D against a query Q sums a per-term score across every query term. For each term, the score multiplies an inverse document frequency weight by a saturated version of the term's frequency in the document. Written out, the score for one term is IDF(qᵢ) · [ f(qᵢ, D) · (k₁ + 1) ] / [ f(qᵢ, D) + k₁ · (1 − b + b · |D|/avgdl) ], and you add those up for all terms in the query.
Each part of the formula maps to a signal:
f(qᵢ, D) is the term frequency: how many times query term qᵢ appears in the document.IDF(qᵢ) is inverse document frequency, which rises for rarer terms.|D| is the length of the document.avgdl is the average length across the collection.k₁ tunes how quickly repeated terms stop contributing to the score.b tunes how strongly length penalizes the score.The three subsections below explain how these pieces interact in practice.
How Does the BM25 Formula Actually Work? The formula's three moving parts: term frequency saturation, inverse document frequency, and length normalization; each solves a specific ranking problem that naive keyword counting cannot. Understanding how they interact is the difference between treating BM25 as a black box and knowing why it produces the rankings it does.
Term Frequency Saturation Prevents Keyword Stuffing From Winning The core of the formula is a saturation function . As a term appears more often, its contribution rises, but with diminishing returns. The f × (k₁ + 1) / (f + k₁ · (...)) shape means the score climbs steeply at first, then flattens toward an asymptote.
A document that mentions a term 500 times is not 100 times more relevant than one that mentions it 5 times. This is the mechanism that stops a page from winning just by repeating a keyword hundreds of times.
Inverse Document Frequency Rewards Rare, Distinctive Terms The IDF component weights each term by its rarity. The canonical form is IDF(qᵢ) = ln( (N − n(qᵢ) + 0.5) / (n(qᵢ) + 0.5) + 1 ), where N is the total number of documents and n(qᵢ) is the number of documents that contain the term. The +0.5 terms guard against division by zero. The +1 inside the logarithm prevents negative IDF, which would otherwise happen when a term appears in more than half the documents.
Lucene BM25Similarity uses a slightly different variant that produces the same effect. Rare terms carry more signal, so a distinctive token, such as an error code, weighs far more than a common word.
Document-Length Normalization Levels the Playing Field The (1 − b + b · |D|/avgdl) factor in the denominator normalizes document length. Without it, longer documents would score higher simply because they contain more words and therefore more term matches. The normalization scales the term-frequency contribution down for documents longer than average and up for shorter ones. Extensions like BM25+ and BM25F build on this base, but the three signals above are the whole story for standard BM25.
How Do You Tune BM25 with k₁ and b? BM25 has two free parameters, and both are worth understanding before you touch them. k₁ controls term-frequency saturation: set it to 0 and term frequency is ignored; raise it, and each repeat adds a meaningful score. b controls document-length normalization: set b to 0 and length has no effect, set it to 1 and you apply full normalization. The table below summarizes the defaults documented by Lucene, Elasticsearch, OpenSearch, and Azure AI Search .
Parameter Default Typical range Effect at the low end Effect at the high end k₁ 1.2 0.5–2.0 Faster saturation; repeats add little Approaches raw term-frequency behavior b 0.75 0.3–0.9 Less length penalty Full-length normalization
Parameter tuning is lower priority than query construction and linguistic controls such as tokenization and stemming. Robertson and Zaragoza found that values around 1.2 < k₁ < 2 and 0.5 < b < 0.8 work reasonably well in many circumstances, so the defaults are a sensible starting point. Hold out a portion of your queries for validation, because parameters that shine on a tuning set can overfit.
With the knobs understood, the harder decision is when BM25 is the right tool at all.
When Should You Use BM25 Instead of Semantic Search? BM25 is a sparse lexical retrieval that matches tokens verbatim, while vector search is a dense semantic retrieval that captures meaning across different wording. The choice comes down to what the query actually contains, and the table below maps common query shapes to the better ranker.
Query characteristic Better fit Why Exact identifiers (SKU, error code, endpoint) BM25 Lexical matching finds verbatim token embeddings dilute Legal or compliance clause numbers BM25 Precise phrase and identifier matching Paraphrased or conceptual questions Semantic Embeddings capture meaning across wording Synonyms and related concepts Semantic BM25 cannot match synonyms Short navigational queries BM25 Lexical precision on rare exact terms Long natural-language questions Semantic Meaning matters more than exact tokens
In practice, BM25 wins when the query contains unique identifiers that appear verbatim in target documents, since embeddings tend to dilute rare tokens. It falls short on paraphrasing because it cannot match synonyms and treats "run," "runs," "running," and "ran" as four different terms unless a stemmer normalizes them first. The production answer is hybrid retrieval, where BM25 is the precision leg and vector search is the semantic leg, fused with a method like reciprocal rank fusion , because directly merging BM25 and vector scores can misrank results when the two systems score evidence on different scales.
Why Does BM25 Still Matter for AI Agents? The tokens agents most need to find in enterprise data are precisely the tokens BM25 handles best. Two properties make it hard to replace: the shape of the data agent's query, and the auditability of the ranking itself.
Enterprise Data Is Full of Exact-Match Tokens Think about what an agent actually queries when it reaches into your business systems. Schema and field names, customer IDs, product SKUs, contract clause numbers, ticket IDs, and error codes are all exact-match targets.
When someone asks an agent to pull the record for customer 4471-A or find the deployment runbook for version 3.2, the answer depends on matching those literal tokens. Semantic embeddings tend to dilute rare identifiers because they encode meaning, and an identifier has no meaning to spread across. Financial documents show the same pattern: company names, metric labels, and fiscal periods give lexical matching strong signals that embeddings can blur.
Interfaces like a Model Context Protocol gateway surface these fields to agents via a governed connection to a unified Context Store, and BM25 is what makes them findable within that connection.
Deterministic Ranking Makes Retrieval Auditable BM25 has three durable strengths for agent systems . It runs entirely on CPU, with no model training and no GPU, so it stays cheap and fast enough to serve as a tool that an agent calls within a reasoning loop. Its ranking is deterministic, and every part of the score has a clear interpretation, so when a result is surprising, you can trace exactly why.
That explainability turns retrieval into an auditable step, which matters when an agent accesses permissioned data and needs to justify what it saw. Teams building through an Agent SDK rely on that transparency to debug retrieval behavior in production.
There is an honest counterpoint. When an agent expands the original query with extra keywords, that expansion can introduce noise if the query terms were already precise matches. On datasets where the original terms were exact, agentic keyword expansion underperformed plain BM25. The lesson is that BM25 is one retrieval primitive an agent selects among others, not a universal answer, and that selection depends on the quality of the corpus underneath.
How Does Airbyte Agents Fit Lexical Retrieval Into a Context Layer? A retrieval primitive is only as good as the corpus it searches. Airbyte Agents is the context layer for AI agents. It connects to the SaaS tools a company runs on and unifies that data into the Context Store, a managed, searchable layer that agents query through Search against the pre-materialized index or directly against the live API.
Because the Context Store pre-materializes typed, unified records behind a single governed auth flow, agents get lexical and semantic retrieval over clean, permission-aware enterprise data, with access enforced and auditable at retrieval time.
Where Does BM25 Fit in Modern Agent Retrieval? BM25 endures not because it is old but because exact-match ranking is what enterprise agent queries most often need. When an agent has to find a specific SKU, ticket ID, or contract clause, lexical precision beats semantic guessing, and the fact that you can trace every score makes it a retrieval step you can trust inside a governed system. That combination, precision plus explainability, is unusual in modern retrieval, and it is why hybrid architectures keep BM25 in the loop rather than replacing it.
That trust depends on the quality of the underlying data, which is where Airbyte Agents come in. It unifies the tools your business runs on into a governed Context Store, with two-mode execution across Search against the pre-materialized index and Direct against the live API. Alongside the Context Store, the Agent MCP, Agent SDK, and Agent CLI provide teams with the interfaces they need to integrate lexical and semantic retrieval into production agents on clean, typed, permissioned records.
Ready to see the governed context in action? Get a demo to explore how your agents can answer real business questions.
Frequently Asked Questions What Does BM25 Stand For? BM25 means "Best Matching 25," the 25th iteration in a series of ranking functions built on the probabilistic retrieval framework developed in the 1970s and 1980s. The "Okapi" prefix, which often appears with it, comes from the Okapi information retrieval system implemented at London's City University in the 1980s and 1990s.
What Is the Difference Between BM25 and TF-IDF? BM25 adds two things TF-IDF lacks. It applies term-frequency saturation, so repeated terms yield diminishing returns rather than linear scaling. It also normalizes for document length, so longer documents are not favored just for containing more words. Both share the same bag-of-words limitation and cannot match synonyms or understand meaning.
What Is BM25F? BM25F is a multi-field extension that treats a document as multiple fields, such as title, body, and tags, each with its own weight. It is useful for structured documents where a match in the title should count more than one buried in a footnote.
Does BM25 Work Without Training or a GPU? Yes. BM25 runs entirely on CPU with no model training and no labeled data. That is why it stays cheap and fast enough to serve as a retrieval tool an agent calls inside a reasoning loop, and why teams reach for it as a baseline before adding heavier neural retrieval.
How Do BM25 and Vector Search Combine in Hybrid Retrieval? Hybrid retrieval runs BM25 and vector search in parallel, then fuses the two ranked lists using a method like reciprocal rank fusion. This avoids the mis-ranking that happens when you naively add the raw scores, since lexical and semantic systems score evidence on different scales.
