Search engines and retrieval systems use two fundamentally different approaches to find relevant content: lexical matching (counting and weighting words) and semantic matching (understanding meaning). Understanding how lexical ranking works explains why terminology choices affect visibility, why keyword stuffing fails, and why modern hybrid systems still rely on term-based signals alongside vector search.
What is lexical ranking?
Lexical ranking scores documents based on the words they contain. A query for "crawl budget optimisation" triggers a search for documents containing those terms; documents with more occurrences, or occurrences of rarer terms, score higher.
This approach dominated search for decades before semantic methods emerged. Google's early algorithms relied heavily on term matching combined with link analysis (PageRank). While semantic search has transformed how systems understand queries, lexical signals remain foundational. Modern search engines and RAG systems run lexical and semantic retrieval in parallel, combining results through rank fusion.
The core question lexical ranking answers: given a query containing specific words, which documents are most likely to be relevant based on the presence and distribution of those words?
Term frequency: the intuition
The simplest relevance signal is term frequency (TF): how often a query term appears in a document. A page mentioning "JavaScript" twenty times seems more likely to be about JavaScript than one mentioning it once.
This intuition has an obvious flaw. A page could mention "JavaScript" hundreds of times while providing no useful information. Early search engines were trivially manipulated by repeating target keywords, a practice that became known as keyword stuffing.
Raw term frequency also fails to account for document length. A 10,000-word article mentioning "JavaScript" fifty times may be less focused on JavaScript than a 500-word article mentioning it ten times. The longer document has more opportunities for any term to appear; raw counts don't reflect concentration.
Term frequency provides a starting point, but needs refinement to produce useful rankings.
Inverse document frequency: rarity as a signal
Some words appear in nearly every document. Terms like "the", "is", "and", and "website" carry little information about what makes a document distinctive. A query for "JavaScript website development" should weight "JavaScript" and "development" more heavily than "website", which appears on most pages about web topics.
Inverse document frequency (IDF) addresses this by weighting terms according to their rarity across the corpus. The formula uses a logarithmic scale: terms appearing in fewer documents receive higher weights.
Consider a corpus of 10 million web pages:
- "the" appears in 9.9 million documents → very low IDF
- "website" appears in 2 million documents → low IDF
- "JavaScript" appears in 500,000 documents → moderate IDF
- "WebAssembly" appears in 50,000 documents → high IDF
- "Binaryen" (a WebAssembly toolchain) appears in 5,000 documents → very high IDF
When someone searches for "Binaryen WebAssembly optimisation", the term "Binaryen" dominates the scoring because its rarity makes it highly discriminative. Documents containing this specific term are far more likely to be relevant than documents merely containing "optimisation".
IDF is computed across the entire corpus, not within individual documents. A term rare in your document but common across the web still receives low IDF weight. This is why domain-specific terminology often outperforms generic phrasing for search visibility.
TF-IDF: combining the signals
TF-IDF multiplies term frequency by inverse document frequency, producing a score that rewards documents containing query terms frequently (TF) while prioritising documents containing rare, discriminative terms (IDF).
A document scores highly when it:
- Contains the query terms (necessary condition)
- Contains those terms multiple times (term frequency)
- Contains terms that are rare across the corpus (inverse document frequency)
This combination addresses keyword stuffing partially. Repeating common words adds little because their IDF is low. Repeating rare terms helps, but only if those terms are genuinely relevant to the query.
What TF-IDF rewards
- Consistent use of target terminology: Documents that use the same term throughout, rather than varying synonyms, accumulate higher term frequency for that specific term.
- Domain-specific vocabulary: Technical terms, product names, and specialised jargon tend to have high IDF because they appear in fewer documents.
- Terms that differentiate: Words that distinguish your content from generic coverage carry more weight than words common to all content on the topic.
What TF-IDF misses
- Synonyms and related concepts: A document about "automobile maintenance" won't match a query for "car repair" despite covering the same topic.
- Query intent: TF-IDF treats all term occurrences equally, regardless of whether the document actually answers the user's question.
- Semantic relationships: The algorithm doesn't understand that "Python" in a programming context differs from "python" in a zoological context.
These limitations motivated the development of semantic search, but they don't make lexical signals obsolete. They explain why both approaches are needed.
How search engines implement lexical ranking
TF-IDF explains what to calculate, but not how to do it efficiently. Scanning billions of documents for every query would be impossibly slow. Search engines solve this through a data structure called an inverted index.
The inverted index
A traditional index maps documents to their contents: "Document A contains words X, Y, Z." An inverted index reverses this relationship: "Word X appears in Documents A, C, F."
For each term in the corpus, the inverted index stores a list of documents containing that term, along with metadata like term frequency and positions. When a query arrives, the search engine looks up each query term in the index and retrieves the corresponding document lists. The intersection and scoring of these lists produces the ranked results.
This lookup is fast because search engines don't scan documents at query time. The hard work happens during indexing, when documents are processed and their terms are added to the inverted index. Query processing becomes a matter of retrieving pre-computed lists and combining them.
Tokenisation and normalisation
Before terms enter the index, they undergo processing:
- Tokenisation splits text into individual terms. "JavaScript frameworks" becomes two tokens: "JavaScript" and "frameworks".
- Case normalisation converts terms to lowercase so "JavaScript" and "javascript" match.
- Stemming or lemmatisation reduces words to root forms. Stemming strips suffixes, so "running" and "runs" index as "run"; lemmatisation goes further, mapping irregular forms like "ran" to "run" as well.
- Stopword handling addresses common words like "the", "is", and "and" that carry little discriminative value. Older systems removed them during indexing to save space; modern systems usually retain them (they matter for phrase queries) and rely on IDF to down-weight them instead.
These transformations affect what queries match which documents. A search for "running" matches documents containing "runs" if the index uses stemming. This preprocessing is one reason lexical search sometimes produces unexpected matches or misses.
Why this matters for content
The inverted index model has practical implications:
- Exact terminology matters: The index stores the processed forms of your words. If your content uses "optimisation" but users search for "optimization", whether they match depends on the search system's normalisation rules.
- Synonyms aren't automatic: Unless the index explicitly maps synonyms during indexing or query expansion, "car" and "automobile" remain separate terms that don't match each other.
- Position data enables phrase matching: Some indexes store word positions, allowing systems to boost documents where query terms appear adjacent to each other. This is how phrase queries like "machine learning" can match differently than two separate terms.
The inverted index is what makes TF-IDF and BM25 practical at web scale. Understanding this structure explains why lexical search behaves the way it does, and why certain content choices affect retrievability.
BM25: the practical evolution
BM25 (Best Matching 25) emerged in the 1990s as a refinement of TF-IDF and has since become the standard lexical ranking algorithm. Elasticsearch, Solr, and most search infrastructure use BM25 by default; Lucene, the library underlying both, adopted it as its default in 2016. Google's systems incorporate BM25-like signals alongside hundreds of other ranking factors.
BM25 addresses two specific weaknesses in raw TF-IDF through tuneable parameters: term saturation and document length normalisation.
Term saturation
In raw TF-IDF, each additional occurrence of a term adds the same amount to the score. A document mentioning "JavaScript" 100 times scores ten times higher than one mentioning it 10 times, all else equal.
This doesn't match intuition about relevance. The difference between 1 and 5 occurrences likely indicates genuine topical focus. The difference between 50 and 100 occurrences probably doesn't. At some point, additional occurrences stop providing evidence of relevance.
BM25 introduces saturation through the k₁ parameter. Term frequency contributions diminish as counts increase, following a curve rather than a line. The first few occurrences contribute substantially; later occurrences contribute marginally.
With a typical k₁ value of 1.2:
- 1 occurrence → baseline contribution
- 5 occurrences → roughly 1.8× the contribution of 1
- 20 occurrences → roughly 2.1× the contribution of 1
- 100 occurrences → roughly 2.2× the contribution of 1 (approaching the maximum)
The curve flattens. Keyword stuffing produces minimal gains because saturation prevents term frequency from scaling linearly.
Document length normalisation
A 5,000-word document naturally contains more term occurrences than a 500-word document, even if both cover the same topic with equal focus. Raw TF-IDF favours longer documents simply because they have more words.
BM25 normalises for document length through the b parameter (typically set around 0.75). The algorithm compares each document's length to the average length in the corpus, adjusting scores so that shorter, focused documents can compete with longer, comprehensive ones.
With standard b values:
- Documents shorter than average receive a scoring boost
- Documents longer than average receive a scoring penalty
- The adjustment is proportional to how far the document deviates from average length
This explains why a concise, focused article often outranks a lengthy, diluted one for specific queries. BM25 rewards concentration of relevant terms rather than absolute counts.
BM25's length normalisation means concise coverage of a narrow topic can outrank comprehensive coverage that dilutes focus. For competitive terms, depth on a specific subtopic often performs better than breadth across many.
Why lexical signals still matter
Semantic search transformed retrieval by matching meaning rather than words. A query about "resetting login credentials" can surface documents about "password recovery" because embedding models recognise the semantic similarity. This solved the vocabulary mismatch problem that plagued pure lexical systems.
However, semantic search introduced new failure modes. Semantic relevance depends on embedding models that encode relationships learned during training. When models haven't learned specific relationships, semantically related content may appear distant in vector space.
This matters most for:
- Product identifiers: "iPhone 15 Pro Max 256GB" contains specific tokens that embedding models may not differentiate from generic smartphone content.
- Technical specifications: Model numbers, version strings, and configuration parameters don't embed distinctively.
- Brand names: Proper nouns often don't carry semantic meaning that embeddings capture reliably.
- Domain-specific terminology: Specialised terms may be underrepresented in training data, producing weak embeddings.
A query for "Bosch GSB18V-490 drill specifications" may retrieve general content about cordless drills if semantic search alone is used. The embedding captures "cordless drill" strongly but treats "GSB18V-490" as noise. Lexical matching finds documents containing that exact string.
Hybrid retrieval as the standard
Production search engines and RAG systems run both retrieval methods in parallel:
- Semantic search: Finds content related by meaning, handling synonyms and conceptual similarity.
- Lexical search (BM25): Finds content containing exact terms, handling specific identifiers and terminology.
Results from both systems merge through rank fusion algorithms (typically Reciprocal Rank Fusion). Content appearing in both result sets ranks highest; content appearing in only one still surfaces if scored highly enough.
Optimising only for semantic relevance misses queries where exact terminology matters. Optimising only for lexical matching misses queries where meaning matters more than specific words. Effective content addresses both.
Practical implications for content
Understanding lexical ranking mechanics translates into specific content practices.
Use terminology consistently
The style guide principle of avoiding "elegant variation" has a technical basis. If your topic is "crawl budget", use "crawl budget" throughout. Switching to "crawl allocation", "indexing capacity", or "crawler resources" for variety means each variant accumulates separate (lower) term frequency scores.
Lexical systems treat each term independently. "Crawl budget" and "crawler resources" are unrelated strings. Semantic systems may recognise them as related, but you're weakening lexical signals unnecessarily.
Consistency also aids reader comprehension. Technical writing repeats canonical terms deliberately; this isn't stylistic weakness.
Include specific identifiers
When content covers specific products, versions, or technical identifiers, include them explicitly:
- Product model numbers (not just product categories)
- Software version strings (not just software names)
- Technical specifications (not just general capabilities)
- Proper nouns and brand names (not just generic descriptions)
These identifiers may be the only terms that distinguish your content from generic coverage. A page about "Elasticsearch 8.x query performance" that never mentions specific version numbers misses queries from users searching for version-specific information.
Balance comprehensiveness with focus
BM25's length normalisation creates a tension between comprehensive coverage and focused relevance. A 10,000-word guide covering twenty subtopics may rank lower than a 1,000-word article covering one subtopic deeply, because the longer document dilutes term concentration.
This doesn't mean shorter is always better. It means structure matters:
- Consider separate pages for distinct subtopics rather than a lengthy page
- Ensure each page has a clear topical focus that accumulates relevant term frequency
- Use internal linking to connect related pages rather than cramming everything into one document
The goal is concentration of relevant terms within documents that match specific queries.
Don't optimise for lexical signals alone
Lexical ranking explains term weighting, not overall search ranking. Modern search engines and AI systems evaluate hundreds of signals: content quality, user engagement, authority, freshness, and semantic relevance alongside term matching.
A page perfectly optimised for BM25 still fails if the content is thin, duplicative, or unhelpful. Lexical signals determine whether your content is found; other signals determine whether it ranks and satisfies users.
Understanding BM25 mechanics doesn't justify keyword-focused content strategies. Beyond BM25's saturation function making keyword stuffing ineffective, search engines maintain explicit spam detection signals that identify and penalise unnatural term repetition. Focus on covering topics thoroughly with appropriate terminology; the lexical signals follow naturally.
FAQs
Is TF-IDF still used in modern search?
Not in its raw form. BM25, which incorporates TF-IDF principles with saturation and length normalisation, is the standard lexical ranking algorithm. Google and other major search engines use BM25-like signals as one component of much larger ranking systems that include semantic understanding, quality signals, and user behaviour data.
How does keyword density relate to TF-IDF?
Keyword density (the percentage of words that are the target keyword) was an early, crude approximation of term frequency relevance. BM25's saturation function makes density largely irrelevant. A 2% keyword density doesn't outrank 1% in any meaningful way because additional term occurrences contribute marginally after the first several. Focus on natural language that covers topics thoroughly rather than targeting density percentages.
Does BM25 handle phrase matching?
Standard BM25 treats queries as bags of words; word order doesn't affect scoring. A query for "machine learning" matches documents containing "machine" and "learning" separately as well as documents containing the phrase. Many search systems add phrase matching as a separate boost on top of BM25 scoring, rewarding documents where query terms appear adjacent or near each other.
How do stopwords affect lexical ranking?
Common words like "the", "is", "and" have extremely low IDF scores and contribute minimally to document rankings. Older systems filtered stopwords during indexing to reduce index size, but modern systems generally retain them and let IDF down-weighting do the work. In practice, whether your content includes or omits stopwords has negligible effect on lexical ranking.
Should I optimise differently for AI search versus traditional search?
The same principles apply. RAG systems use BM25 or similar lexical algorithms as part of hybrid retrieval. The difference is that AI systems may weight semantic relevance more heavily and evaluate information gain differently. Content that works well for both: specific terminology for lexical matching, clear semantic focus for embedding quality, and substantive information that provides value when retrieved.
Key takeaways
- TF-IDF: Documents score higher when they contain query terms frequently, especially terms that are rare across the corpus.
- BM25: Diminishing returns on term frequency prevent keyword stuffing; length normalisation lets focused content compete with comprehensive coverage.
- Hybrid retrieval: Modern systems use both lexical and semantic approaches; exact terminology matters for identifiers and domain-specific terms that don't embed distinctively.
- Terminology consistency: Using canonical terms throughout a document accumulates stronger term frequency than varying synonyms.
- Practical focus: Search engines and RAG systems combine BM25 with vector search; content must perform well in both to maximise visibility.
Further reading
- Okapi BM25 (Wikipedia)
Technical overview of the BM25 ranking function, its mathematical formulation, and parameter tuning - Practical BM25: The BM25 Algorithm and Its Variables
Elasticsearch's accessible explanation of how k₁ and b parameters affect ranking behaviour - TF-IDF Weighting (Stanford NLP)
Academic treatment of term frequency and inverse document frequency from the Stanford NLP group